Sync eng/common directory with azure-sdk-tools for PR 16727 - #48596
Merged
azure-sdk-automation[bot] merged 1 commit intoAug 19, 2026
Merged
Conversation
|
Azure Pipelines: Successfully started running 1 pipeline(s). 9 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot could not run the full agentic suite for this review because it was automatically requested on a bot-authored pull request. Request a review from Copilot under Reviewers to retry with the full agentic suite. Improved support for bot-authored pull requests is coming soon.
Adds an opt-out switch to skip environment setup during test resource deployment, enabling faster or more targeted runs.
Changes:
- Introduces a new
SkipEnvironmentSetuptemplate parameter with a default offalse. - Wraps the
setup-environments.ymltemplate step in a conditional so it can be skipped.
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
azure-sdk-automation
Bot
force-pushed
the
sync-eng-common-AllowAzModulesAndDogfoodSkip-16727
branch
from
August 18, 2026 19:33
1d04e6e to
772cd4f
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.
Suppressed comments (1)
eng/common/TestResources/deploy-test-resources.yml:46
- For boolean parameters, prefer using
not(parameters.SkipEnvironmentSetup)(or an equivalent negation form) instead ofeq(..., false). This is typically clearer and avoids subtle issues if the parameter value is ever supplied in a non-boolean form.
- ${{ if eq(parameters.SkipEnvironmentSetup, false) }}:
- template: /eng/common/TestResources/setup-environments.yml
Mike Harder (mikeharder)
approved these changes
Aug 18, 2026
azure-sdk-automation
Bot
deleted the
sync-eng-common-AllowAzModulesAndDogfoodSkip-16727
branch
August 19, 2026 19:16
Howie Leung (howieleung)
added a commit
that referenced
this pull request
Aug 20, 2026
* [Service Bus] Send server-timeout on management operations (#48563) Management operations (peek, deferred receive, settlement over the management link, lock renewal, session state, session listing, schedule/cancel scheduled) now send `com.microsoft:server-timeout`: the caller's remaining time less a one second buffer, or 60 seconds when none was given, floored at zero and capped at the AMQP uint maximum. Previously no bound was sent, so a stalled service held the call until the AMQP link failed; it now raises a retryable `OperationTimeoutError`, surfacing after roughly four minutes at default retries. A settle over the management link can therefore raise `MessageLockLostError` where the call used to block. The data path is unchanged: `receive_messages`, receiver iteration and `send_messages` are unaffected, as is CBS token auth. Set in the base handler after the retry loop computes the remaining time, so it reflects the time left on the attempt. Both transports pass application properties through unchanged, so no transport change is needed. `REQUEST_RESPONSE_TIMEOUT` already held the correct key and was unused. Tests cover the arithmetic and its boundaries, that the property reaches the outgoing message, that the service's answer surfaces as a retryable `OperationTimeoutError`, and that the value carries the transport's own type. api.metadata.yml carries a gated parserVersion bump only; apiMdSha256 is unchanged, so no public API moved. Matches the .NET, Java and Go SDKs. Refs: AB#38822503 Co-authored-by: Pranjal Patel <pranjalpatel@microsoft.com> * Adding fix workflow to failing pipelines (#48554) * Adding a new github actions workflow to trigger on pipeline failures. * Updated copilot feedback. * Update .github/workflows/pipeline-analysis-next-steps.md Co-authored-by: Daniel Jurek <djurek@microsoft.com> * Updated with feedback. * Add temporary test pipeline for gh-aw trigger validation * Revert "Add temporary test pipeline for gh-aw trigger validation" This reverts commit 12764ed. * . (#48155) * Delete test-trigger-pipeline.yml * Removed manual runs from trigger. Updated comments * Updating the analysis workflow and adding the fix workflow. Also added branch cleanup connected to the agentics-maintenance workflow. * Removed testing checks. * Updated concurrency issues and multiple suite runs. * Updated fix workflow steps with more detail. * Fixed edge cases and broadened branch cleanup. * Implemented feedback. * Removed branch cleanup. Updated analysis workflow to exclude fix PRs. * Switched to azsdk cli to satisfy copilot feedback. * Addressed Copilot feedback. --------- Co-authored-by: Daniel Jurek <djurek@microsoft.com> * Re-enable recorded azure-ai-ml e2e tests (#48538) * Re-enable recorded azure-ai-ml e2e tests Refresh sanitized test-proxy recordings, restore playback coverage for datastore, connection, environment, DSL, and pipeline e2e cases, and update parallel test environments for supported runtimes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: fd9e755b-1b9d-4918-aa65-1774f7f78014 * Fix azure-ai-ml recording placeholder mismatch Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: fd9e755b-1b9d-4918-aa65-1774f7f78014 * Fix azure-ai-ml playback sanitizer matching Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: fd9e755b-1b9d-4918-aa65-1774f7f78014 * Refresh remaining azure-ai-ml recordings Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: fd9e755b-1b9d-4918-aa65-1774f7f78014 --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: fd9e755b-1b9d-4918-aa65-1774f7f78014 * [Service Bus] Make PEEK_LOCK settlement outcomes observable on pyamqp PEEK_LOCK receiver links negotiate `rcv-settle-mode=second`, meaning settle only once the service confirms the outcome, but the SDK sent every disposition pre-settled and inherited a no-op `_incoming_disposition`. The service's terminal outcome was never requested and would have been discarded anyway, so `complete_message()` and its siblings could not fail: a settlement the service never applied returned success in ~0.4 ms without a round trip, and rejections carrying `com.microsoft:message-lock-lost` or `com.microsoft:session-lock-lost` were dropped. The only symptom was the message reappearing at lock expiry with an incremented delivery count. Verified live: for the same dead lock in the same session, the receiver link reported success in 0.42 ms while the management link raised SessionLockLostError and the message stayed queued. Settlements are now sent unsettled and wait for the service to confirm, raising when it rejects and reporting an unconfirmed outcome so the existing management-link fallback can re-settle authoritatively. A disposition only confirms when the frame is settled; an outcome alone does not, matching SenderLink. This is derived rather than configured, since .NET, Java, JavaScript and Go all await the disposition unconditionally and none exposes a flag to disable it. It is skipped only where there is no outcome to observe: RECEIVE_AND_DELETE is settled by the service on delivery, and uamqp cannot report dispositions. Success compares the echoed outcome against the one requested, because the service echoes what it applied rather than always replying accepted (abandon and defer echo modified, dead_letter echoes rejected); requiring accepted would have failed three of the four operations. This is a behavior change: confirming costs one round trip per settlement (~1-3 ms in region), so settle concurrently rather than serially, for example `await asyncio.gather(*(receiver.complete_message(m) for m in messages))`. The public surface is untouched and api.md is byte-identical to main. Also fixes a catch-all `except AMQPException` that flattened the service's real error condition into a generic ServiceBusConnectionError, hiding message-lock-lost from callers, and documents the symptom in TROUBLESHOOTING.md — a message redelivered despite a successful complete_message() previously had no entry point. * Allow test resource deployment to skip environment setup (#48596) Co-authored-by: alzimmermsft <48699787+alzimmermsft@users.noreply.github.com> * Add npm userconfig to typespec emitter template (#48639) Fixed CFSClean Co-authored-by: Ray Chen <raychen@microsoft.com> * Pipeline analysis and fix patch (#48643) * Fix pipeline analysis pull request resolution * Fix check suite repository matching * Fix pipeline auto-fix comment status * Address pipeline workflow review feedback * [CI] Remove api-consistency check (#48638) * Remove api-consistency check. * Restore delted api and metadata files. * Revert eventgrid api.metadata.yml to original content Co-authored-by: tjprescott <5723682+tjprescott@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: tjprescott <5723682+tjprescott@users.noreply.github.com> * [CI] Generate ApiHash during extended build (#48628) * Refactoring, and add apiHash generation to build-extended-artifacts.yml * Code review feedback. * Test change for azure.template * Revert azure-template changes (except pyproject.toml) * Pass PI consistency... * [Service Bus] Clarify application_properties returns bytes keys/values on receive (#48078) * docs: Clarify application_properties returns bytes keys on receive (#45082) - Correct the ServiceBusMessage constructor :paramtype for application_properties from Dict[str, ...] to Dict[Union[str, bytes], ...] to match the existing parameter annotation - Add a note on the application_properties property that, when a message is received, keys and string values are returned as bytes, with the bytes-key access and decode pattern; note the same applies to the raw AMQP annotations and delivery_annotations accessed via raw_amqp_message - Add a README sample for reading application properties from received messages - Add a CHANGELOG Other Changes entry Documentation-only change; no behavioral change. * docs: Address review feedback on application_properties note (#45082) - Guard the Optional application_properties against None before calling .get in both the docstring and README examples, so copying the sample does not raise AttributeError on a message that carries no application properties - Correct the value-type note: an AMQP timestamp decodes to an integer (milliseconds), not a datetime; list only the verified native types (int, bool, float, uuid.UUID) and reframe as an AMQP-decode description * eng: Refresh Service Bus API metadata * Release agentserver responses 2.1.0b2 (#48653) Move post-2.1.0b1 fixes into a new release entry dated 2026-08-20 and bump the package version. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: bdc32ee9-9e8f-4a1c-ba27-bdfa318afa1e * [agentserver-responses] Keep newest history item IDs when applying limit (#48560) * Keep newest history item IDs when applying limit get_history_item_ids in the in-memory and file response stores used resolved[:limit], which dropped recent turns. Slice from the end so conversation history keeps the newest N items. Fixes #48514. * Keep newest history ids under the limit and fix changelog * Restore an unreleased changelog section for the history id limit fix. Keep 2.1.0b1 as previously released and name FileResponseStore in the 2.1.0b2 notes. * Bump azure-ai-agentserver-responses to 2.1.0b2 so Analyze validates the unreleased changelog. * Clarify that limit keeps newest item IDs from the resolved chain. --------- Co-authored-by: Shiva S <shivakishore14@gmail.com> * docs(servicebus): clarify and sample session listing (#48645) * Fix Challenge Auth replay bug in Keyvault Security Domain SDK (#48636) * Ported fix * Updated changelog * generated Api.md for the package as its a new CI requirement --------- Co-authored-by: Hari K <kha@microsoft.com> * updated changelog (#48657) Co-authored-by: Hari K <kha@microsoft.com> * [AutoPR azure-mgmt-compute-bulkaction]-generated-from-SDK Generation - Python-6698911 (#48575) * Configurations: 'specification/compute/resource-manager/Microsoft.Compute/Bulkactions/tspconfig.yaml', SDK Release Type: beta, and CommitSHA: 'da7c67564bd3344610ffb0ec21a1974e79ffb00a' in SpecRepo: 'https://github.com/Azure/azure-rest-api-specs' Pipeline run: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=6698911 Refer to https://eng.ms/docs/products/azure-developer-experience/develop/sdk-release/sdk-release-prerequisites to prepare for SDK release. * update Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3f1ab584-1bba-44a3-ac54-43dee4f2b16a * Update CHANGELOG.md --------- Co-authored-by: azure-sdk <azuresdk@microsoft.com> Co-authored-by: ChenxiJiang333 <v-chenjiang@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: ChenxiJiang333 <119990644+ChenxiJiang333@users.noreply.github.com> Co-authored-by: Yuchao Yan <yuchaoyan@microsoft.com> Copilot-Session: 3f1ab584-1bba-44a3-ac54-43dee4f2b16a * [AutoPR azure-mgmt-chaos]-generated-from-SDK Generation - Python-6627851 (#48301) * Configurations: 'specification/chaos/resource-manager/Microsoft.Chaos/Chaos/tspconfig.yaml', SDK Release Type: beta, and CommitSHA: '358882ad707dcc3b0d1de9df6ba4dfba7e85da1a' in SpecRepo: 'https://github.com/Azure/azure-rest-api-specs' Pipeline run: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=6627851 Refer to https://eng.ms/docs/products/azure-developer-experience/develop/sdk-release/sdk-release-prerequisites to prepare for SDK release. * update * Update CHANGELOG.md --------- Co-authored-by: azure-sdk <azuresdk@microsoft.com> Co-authored-by: ChenxiJiang333 <v-chenjiang@microsoft.com> Co-authored-by: ChenxiJiang333 <119990644+ChenxiJiang333@users.noreply.github.com> * Howie/httpx2 (#48651) * Update version * Restore tsp-location.yaml * Update emitter skill for Python to deal with prerequisites (#47878) * change log (#47916) * [azure-ai-projects] Emit SDK from TypeSpec (commit fca510e0) (#47914) * Fix TypeSpec paths (#47959) * Remove sample_agent_toolbox_skill.py per bakcned folks and Linda request, add new hosted agent samples for Teams message trigger and reminder preview (#48234) * Remove sample_agent_toolbox_skill.py per bakcned folks and Linda request, add new hosted agent samples for Teams message trigger and reminder preview * change log * Re-emit from latest TypeSpec and do required updates (#48216) * Re-emit, to remove WebIQ tools (#48240) * update report * Updates in prep for a release of 2.4.0 (#48248) * Update rename tsp-location.yaml, so it does not break release build * change log (#48246) * Add sample toolboxes for synchronous and asynchronous AIProjectClient usage; update assets.json tag * Comment out additional sample tests in TestSamples class * Update to version 2.5.0 * Howie/sample 35 (#48310) * Refactor agent name retrieval to use a fallback mechanism - Updated multiple sample scripts to change the way the agent name is retrieved from environment variables. - Replaced the default value assignment using `os.environ.get("FOUNDRY_AGENT_NAME", "MyAgent")` with a more concise approach using `os.environ.get("FOUNDRY_AGENT_NAME") or "MyAgent"`. - This change ensures that if the environment variable is not set, the fallback value "MyAgent" is still used, while improving code readability. - The affected files include various agent tools and hosted agent samples across the project. * rever dataset generation job polling and update assistant prompt * change log (#48427) * Custom LRO pollers to enable easy access to Job ID (#48468) * Re-emit from latest TypeSpec commit (8-10-2026 17:16) (#48526) * update api.md files * Update public methods report. Delete other old report * Howie/reemit (#48608) * Add A2A protocol support and update related models - Introduced A2ATool and A2AToolboxTool classes for A2A protocol implementation. - Added A2AProtocolVersion enum to define supported A2A protocol versions. - Updated DataGenerationJobOptions to include SimulationSeedDataGenerationJobOptions and removed TaskGenerationDataGenerationJobOptions. - Modified existing models and enums to accommodate new A2A features. - Updated sample agent to utilize the new A2ATool class. - Adjusted beta routines API to remove unsupported parameters and enhance pagination handling. - Updated YAML configuration to reflect repository changes. * Update tsp-location.yaml with latest commit and repository details * Update sample_agent_to_agent.py skip reason in test_samples.py * Update CHANGELOG.md to include new A2A tools and breaking changes (#48626) * Restore tsp-location.yaml file with original content * Update API metadata, fix README A2A preview label, and adjust package lock files * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Rename a2_a_version to a2a_version for consistency in A2A tool classes and code re-emit * Refactor upload_session_file method and improve type hints - Updated the `build_agents_upload_session_file_request` function to handle optional content type. - Enhanced the `upload_session_file` method in `AgentsOperations` class to support both bytes and IO[bytes] types for content. - Added overloads for `upload_session_file` to clarify method usage and parameters. - Improved docstrings for better clarity on parameters and return types. - Fixed a typo in the sample agent to agent script. - Updated tsp-location.yaml with the latest commit hash and formatted additional directories for better readability. * change log * Update API metadata and improve type hints for upload_session_file method * Add httpx to development requirements * Update httpx version constraint in development requirements * Add OpenAI v3 transport compatibility for httpx2 in sample executor * version update * Update OpenAI client to use httpx2 and raise minimum openai package version to 3.0.0 * Add cache_write_tokens to input_tokens_details in MemoryStoreOperationUsage * Update Python version requirement to 3.10 and remove deprecated Python 3.9 classifier * Update CHANGELOG for dependency changes and minimum Python version --------- Co-authored-by: Darren Cohen <39422044+dargilco@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Enable bare messages= input on Relevance/Similarity/Fluency/Retrieval/ResponseCompleteness + all RAI evaluators (#48629) * Support bare messages= input on Relevance + Similarity evaluators Both evaluators previously required either a conversation dict or scalar query/ esponse inputs. When the SDK batch engine invokes them from a customer's data_mapping: {"messages": "${data.messages}"} (produced by ACA on unified-messages evaluation groups), it sends a bare messages=[...] kwarg alongside optional scalar context, ground_truth, and ool_definitions kwargs. The old code paths rejected this combination with: RelevanceEvaluator: "Either 'conversation' or individual inputs must be provided." ("No data to process" family) SimilarityEvaluator: "'query' is missing" / "Cannot provide both 'conversation' and individual inputs at the same time." Fix: - Swap ConversationValidator for MessagesOrQueryResponseInputValidator in each evaluator's __init__ so validation accepts either shape. - Add a _convert_kwargs_to_eval_input override in each evaluator that hoists a bare messages=[...] kwarg into conversation={"messages": ...} and merges any top-level context / ool_definitions scalars into the same conversation dict. A top-level ground_truth scalar is stamped onto each assistant turn that doesn't already carry one, so the base _derive_conversation_converter picks it up per-response. Regression paths preserved: - Passing conversation={"messages": [...]} still works (unchanged). - Passing scalar query/ esponse(/ground_truth) still works. - SimilarityEvaluator with query=None still rejects with the pre-existing error message. Verified end-to-end against real Azure OpenAI (gpt-4o-mini): - Relevance with bare messages -> score 5.0, pass. - Similarity with bare messages + per-turn ground_truth -> score 5.0, pass. - Similarity with bare messages + top-level context + ground_truth + tool_definitions -> score 5.0, pass. - Relevance/Similarity with q/r scalars -> score 5.0, pass (regression). - Similarity with query=None -> correctly rejected (regression). This is a partial migration; the same pattern applies to the other prompty/metric/RAI/tool evaluators that today reject messages-format input at turn/conversation level. * Extract hoist_messages_to_conversation helper for shared reuse The _convert_kwargs_to_eval_input normalization added to RelevanceEvaluator and SimilarityEvaluator was identical logic in two places. Extracted it into _evaluators/_common/_conversation_input.py as hoist_messages_to_conversation so: - The other ~21 bucket-3/4 evaluators (fluency, intent_resolution, response_completeness, retrieval, tool_call_*, RAI safety family, metric family) can adopt the same pattern with a ~3-line override each, instead of copying the same 20-line block. - The normalization has a single documented contract about how bare messages= + scalar context/ground_truth/ ool_definitions kwargs get folded into conversation={...}. Net line count in this commit: -39 (removed inline blocks, added helper + imports). Re-verified end-to-end with real gpt-4o-mini scoring — all 6 scenarios (bare messages, bare messages + adjuncts on both evaluators, q/r baselines, query=None regression) behave identically to the pre-extraction state. * Support bare messages= input on Fluency, Retrieval, ResponseCompleteness Apply the shared hoist_messages_to_conversation helper pattern to the remaining prompty-family evaluators with a conversation overload: - FluencyEvaluator: validator swap ConversationValidator -> MessagesOrQueryResponseInputValidator + convert override - RetrievalEvaluator: convert override (no explicit validator to swap) - ResponseCompletenessEvaluator: convert override (no explicit validator) Not included in this commit: IntentResolutionEvaluator. Its __call__ signature only exposes one overload (query: Union[str, List[dict]], response: Union[str, List[dict]], tool_definitions) with NO conversation overload, so hoisting messages=[...] into conversation={...} does not help — the base singleton matcher never routes into a conversation converter. That evaluator needs a different pattern (either a new conversation overload OR a custom transform that splits the messages list into a leading query context + trailing response turns per the intent-resolution semantic). Deferred to a follow-up. Verified end-to-end against real gpt-4o-mini: - Fluency messages input -> score 2.0 (baseline response scalar -> 3.0); no more "No data to process" error, evaluator runs to completion. - Retrieval messages+context input -> reaches the model; both messages and baseline paths return NaN, so any NaN behavior is not tied to the migration (baseline had it too before the change). - ResponseCompleteness messages+ground_truth input -> score 3.0 pass, identical to baseline. * Support bare messages= input on all RAI service evaluators Hoist a bare `messages=[...]` kwarg (plus optional scalar `context` / `ground_truth` / `tool_definitions`) into `conversation={...}` inside `RaiServiceEvaluatorBase._convert_kwargs_to_eval_input` so every RAI safety evaluator inherits messages-input support in one change instead of eight: - ViolenceEvaluator - HateUnfairnessEvaluator - SelfHarmEvaluator - SexualEvaluator - ProtectedMaterialEvaluator - IndirectAttackEvaluator (XPIA) - CodeVulnerabilityEvaluator - ECIEvaluator All 7 publicly-exported RAI evaluators verified to inherit the hoist via MRO inspection. The base class already routes `conversation` into `_evaluate_conversation` which per-turn-splits and calls the RAI service, so no service-side change is required — the hoist just lets the customer's `messages=` kwarg reach that existing path instead of failing at the kwarg matcher. Regression: pre-existing `query`/`response` scalar calls are untouched (hoist is a no-op when `messages` is absent). Legacy-endpoint path (`_use_legacy_endpoint`) still short-circuits with the raw conversation dict when supplied directly. Not exercised end-to-end here: firing a live RAI service call requires an Azure AI project scope with RAI credentials, which is a separate infra concern from the SDK plumbing. The kwarg-level fix (verified above) is what the customer-facing "No data to process" error was tied to. * Black: collapse assistant-turn conditional into one line * Pin openai<3.0 to unblock CI (test-proxy httpx/httpx2 mismatch) openai released v3.0 which migrated its transport layer from httpx to httpx2. The eval SDK test proxy (tests/__openai_patcher.py) still constructs httpx.URL objects, so with openai 3.x the tests fail with: TypeError: Invalid type for url. Expected str or httpx.URL, got <class 'httpx2.URL'> Verified empirically: the passing PR CI run (build 6717764, queued 01:15 UTC) resolved openai==2.54.0 with httpx; the failing run (build 6717844, queued 01:50 UTC) resolved openai==3.0.0 with httpx2. The only relevant change between the two runs was that openai 3.0 was published to PyPI in that ~35 min window, and pip picked it because setup.py had no upper bound. Pinning openai<3.0 restores the CI to the passing configuration. When the test proxy is updated to use httpx2 (or a compatibility shim), the pin should be lifted. This does not affect the messages= input feature under review; it is strictly a CI unblocker. * Address Copilot review: mutation, docs, and unit tests Addresses three of the four Copilot review findings on the hoist_messages_to_conversation helper: Finding 2 (real bug): the helper mutated caller-provided message dicts in place when stamping a top-level ground_truth. Two evaluations reusing the same messages list with different ground_truth values would see the first call's value leak into subsequent calls. Fix: shallow-copy the messages list and shallow-copy each affected assistant-turn dict before injecting ground_truth. Zero runtime cost, purely allocation. Finding 1 (edge case): explicitly document behavior when messages entries are not plain dicts (typed Message objects, dataclasses). The helper skips ground_truth stamping for such items but still hoists the overall input into conversation. The base `_derive_conversation_converter` requires plain dicts and would fail on non-dicts regardless, so this matches existing SDK convention. Finding 4 (test coverage): add tests/unittests/test_conversation_input.py with seven table-driven tests covering: - no messages, no conversation (noop) - conversation already provided (short-circuit) - bare messages only - bare messages + all adjuncts (context, ground_truth, tool_definitions) - non-dict messages entries skipped for stamping - immutability: same messages list reused across two calls with different ground_truth scalars (regression guard for Finding 2) - per-turn ground_truth is not overwritten by top-level scalar Verified end-to-end with live gpt-4o-mini scoring on all 5 direct-file-edit evaluators (Relevance, Similarity, Fluency, Retrieval, ResponseCompleteness). All previously scored scenarios still score identically; the new immutability guarantee is validated at runtime by reusing the same messages list across two consecutive Similarity calls with different ground_truth values and asserting the caller's assistant dict is unchanged after both. Finding 3 (clearer error for mixed messages+q/r inputs) is not addressed here: the base class already rejects the mixed combination correctly, just with a generic error message. Cosmetic; deferred. * Move openai<3.0 pin from install_requires to dev_requirements Addresses reviewer feedback: the previous `openai<3.0` upper bound on install_requires punished end-user installs by making them incompatible with the current openai 3.x line (3.0 released 2026-08-12, latest 3.3 released 2026-08-18) even though the SDK's runtime is fine with openai 3.x. The incompatibility is confined to the CI test proxy: tests/__openai_patcher.py:74 request.url = httpx.URL(config.proxy_url).join(request_path) openai 3.x switched its transport layer to httpx2 and rejects httpx.URL objects with: TypeError: Invalid type for url. Expected str or httpx.URL, got <class 'httpx2.URL'> Production code paths that use `AsyncAzureOpenAI` directly never reach that patcher, so end users installing this SDK alongside openai 3.x are not affected. Revert the setup.py upper bound and move the constraint to dev_requirements.txt so only the test environment is pinned. Removes this pin once tests/__openai_patcher.py is updated for httpx2 compatibility (out of scope for this PR). * Black: collapse multi-line if in ground_truth stamping loop * Add wiring tests for bare messages= input on the 5 direct-edit evaluators Extends test_conversation_input.py from 7 to 25 tests. The existing 7 cover the hoist_messages_to_conversation helper in isolation. The new 18 prove each of the 5 evaluators overridden in this PR actually calls the hoist and that the hoisted output equals the equivalent conversation={"messages": ...} invocation. Coverage matrix: - A. Bare messages=[...] (5 evaluators) - B. messages= + context + ground_truth + tool_definitions (5 evaluators) - C. Legacy (query, response, ...) regression net (5 evaluators) - D. Negative wiring: empty list, mixed query+messages, non-dict message entries (3 tests) Assertion strategy: shape equivalence with the conversation={...} call path. Robust to base-class output-shape changes because both sides re-execute in lockstep. 25 passed, 0 failed locally (Python 3.10). * Add wiring tests for all 8 RAI evaluators fixed via RaiServiceEvaluatorBase Extends test_conversation_input.py from 25 to 51 tests. Every one of the 13 evaluators fixed by this PR now has direct wiring coverage. RAI evaluators covered (parametrised over all 8): - ViolenceEvaluator - HateUnfairnessEvaluator - SelfHarmEvaluator - SexualEvaluator - ProtectedMaterialEvaluator - IndirectAttackEvaluator (XPIA) - CodeVulnerabilityEvaluator - ECIEvaluator Coverage matrix per RAI evaluator (24 tests): - A. Bare messages=[...] - B. messages= + context + ground_truth + tool_definitions - C. Legacy (query, response) regression net Plus 2 branch tests for the RAI base override's short-circuit: - _use_legacy_endpoint=True returns [kwargs] intact with hoisted conversation (single-call pre-sync-migration behavior). - _use_legacy_endpoint=False delegates to super()'s per-turn converter and does not pass raw messages= kwarg through. Same assertion strategy as the 5 direct-edit evaluators: shape equivalence with the equivalent conversation={...} invocation. 51 passed, 0 failed locally (Python 3.10). * Apply Black formatting to test_conversation_input.py (line-length 120) * Update tsp-location.yaml to change commit hash and format additional directories as a list * Update package lock and JSON files; fix variable naming in models and update commit hash in tsp-location.yaml * update version --------- Co-authored-by: Pranjal Patel <patelpranzp25@gmail.com> Co-authored-by: Pranjal Patel <pranjalpatel@microsoft.com> Co-authored-by: ReilleyMilne <91291100+ReilleyMilne@users.noreply.github.com> Co-authored-by: Daniel Jurek <djurek@microsoft.com> Co-authored-by: lavakumarrepala <v-rlava@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: azure-sdk-automation[bot] <191533747+azure-sdk-automation[bot]@users.noreply.github.com> Co-authored-by: alzimmermsft <48699787+alzimmermsft@users.noreply.github.com> Co-authored-by: Ray Chen <raychen@microsoft.com> Co-authored-by: Travis Prescott <tjprescott@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: tjprescott <5723682+tjprescott@users.noreply.github.com> Co-authored-by: Eldert Grootenboer <eldert@eldert.net> Co-authored-by: Shanmukha Pasumarthy <shanmukha98@gmail.com> Co-authored-by: Hashim Khan <64767361+Hashim1999164@users.noreply.github.com> Co-authored-by: Shiva S <shivakishore14@gmail.com> Co-authored-by: Hariharan K <reachk.hariharan@gmail.com> Co-authored-by: Hari K <kha@microsoft.com> Co-authored-by: azure-sdk <azuresdk@microsoft.com> Co-authored-by: ChenxiJiang333 <v-chenjiang@microsoft.com> Co-authored-by: ChenxiJiang333 <119990644+ChenxiJiang333@users.noreply.github.com> Co-authored-by: Yuchao Yan <yuchaoyan@microsoft.com> Co-authored-by: Darren Cohen <39422044+dargilco@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Rabah B <134166186+RabsB@users.noreply.github.com> Copilot-Session: fd9e755b-1b9d-4918-aa65-1774f7f78014 Copilot-Session: bdc32ee9-9e8f-4a1c-ba27-bdfa318afa1e Copilot-Session: 3f1ab584-1bba-44a3-ac54-43dee4f2b16a
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sync eng/common directory with azure-sdk-tools for PR Azure/azure-sdk-tools#16727 See eng/common workflow