TAPAttack attack execution error

[3leg0ocat]

from pyrit.executor.attack import AttackExecutor

AttackExecutor().execute_multi_turn_attacks_async(TAPAttack())

The code above will throw an error when run:
Attack strategy TreeOfAttacksWithPruningAttack must use MultiTurnAttackContext or a subclass of it.

# Validate that the attack uses MultiTurnAttackContext
if hasattr(attack, "_context_type") and not issubclass(attack._context_type, MultiTurnAttackContext):
    raise TypeError(
        f"Attack strategy {attack.__class__.__name__} must use MultiTurnAttackContext or a subclass of it."
    )

I'd like to know if this design intentionally excludes TAP? Could the type checking be relaxed to allow TAPAttackContext as well?

[romanlutz]

@3leg0ocat did you figure it out?

[3leg0ocat]

@romanlutz Thanks, that part is clear now. However, there is still a related issue that consistently reproduces when using TAPAttack together with scorers other than the simple example ones (e.g. SelfAskCategoryScorer).

The error occurs when printing the attack result stage:
ConsoleAttackResultPrinter().print_result_async()
It fails inside group_conversation_message_pieces_by_sequence with:

ValueError: All message pieces must be from the same conversation.
Expected conversation_id='...', but found '...'.

This indicates that message_pieces contains entries from multiple conversation_ids.

The failure happens when querying memory entries by conversation_id with join_scores=True:

memory_entries: Sequence[PromptMemoryEntry] = self._query_entries(
    PromptMemoryEntry,
    conditions=and_(*conditions) if conditions else None,
    join_scores=True,
)
message_pieces = [memory_entry.get_message_piece() for memory_entry in memory_entries]
return sort_message_pieces(message_pieces=message_pieces)

Although the query is filtered by conversation_id, additional entries seem to be included.
Looking at the query implementation, this seems to be the relevant logic:

query = session.query(Model)
if join_scores and Model == PromptMemoryEntry:
    query = query.options(joinedload(PromptMemoryEntry.scores))
if conditions is not None:
    query = query.filter(conditions)
return query.all()

Based on this, it seems that enabling join_scores=True may change what gets loaded alongside the filtered entries. Could this also be including message_pieces from the scoring stages?

I also compared SelfAskCategoryScorer with FloatScaleThresholdScorer, and one
notable difference is that SelfAskCategoryScorer passes attack_identifier into _score_value_with_llm, for example:

unvalidated_score = await self._score_value_with_llm(
    ...
    scored_prompt_id=message_piece.id,
    objective=objective,
    attack_identifier=message_piece.attack_identifier,
)

Initially I suspected this might affect how scoring records are associated or later retrieved. However, even after explicitly setting
attack_identifier=None, the same conversation mismatch error still occurs.

Given this, could you advise on the recommended way to handle this?

Is SelfAskCategoryScorer expected to be incompatible with TAPAttack, or is
there a supported workaround or configuration to use it without hitting the
conversation mismatch error during result printing?

[romanlutz]

Reopening this because it requires some thought and perhaps an investigation.