Adding Microsoft SECURITY.MD - #2
Merged
Gaurav Sharma (bewithgaurav) merged 1 commit intoJan 24, 2025
Merged
Gaurav Sharma (bewithgaurav) merged 1 commit into
Gaurav Sharma (bewithgaurav) merged 1 commit into
Conversation
Gaurav Sharma (bewithgaurav)
added a commit
that referenced
this pull request
Nov 10, 2025
- Replace pybind11 row[col-1] = value with PyLong_FromLong + PyList_SET_ITEM - Applies to SQL_INTEGER, SQL_SMALLINT, SQL_BIGINT - Eliminates pybind11 wrapper overhead, bounds checking, and extra reference counting - Expected improvement: ~40-100ms for integer-heavy result sets (1.2M rows) - Added PERF_TIMER for int_c_api_assign, smallint_c_api_assign, bigint_c_api_assign - PyList_SET_ITEM steals reference (no Py_INCREF needed)
Gaurav Sharma (bewithgaurav)
added a commit
that referenced
this pull request
Nov 10, 2025
…ypes - Replaced pybind11 wrappers with direct Python C API calls - SQL_INTEGER, SQL_SMALLINT, SQL_BIGINT: PyLong_FromLong/PyLong_FromLongLong - SQL_TINYINT, SQL_BIT: PyLong_FromLong/PyBool_FromLong - SQL_REAL, SQL_DOUBLE, SQL_FLOAT: PyFloat_FromDouble - Uses PyList_SET_ITEM macro for direct list assignment (no bounds checking) - Eliminates pybind11 wrapper overhead for simple numeric types - Added PERF_TIMER instrumentation for each numeric type conversion
Gaurav Sharma (bewithgaurav)
added a commit
that referenced
this pull request
Nov 10, 2025
- Created typedef ColumnProcessor for function pointer type - Added ColumnProcessors namespace with specialized inline processors: * ProcessInteger, ProcessSmallInt, ProcessBigInt, ProcessTinyInt, ProcessBit * ProcessReal, ProcessDouble * ProcessChar, ProcessWChar, ProcessBinary (handle LOBs, NULL, zero-length) - Added ColumnInfoExt struct to pass metadata efficiently - Build columnProcessors array once during cache_column_metadata - Fast path: Direct function call via columnProcessors[col-1] (no switch) - Slow path: Fallback switch for complex types (DECIMAL, DATETIME, GUID) - Eliminates switch evaluation from O(rows × columns) to O(columns) - All processors use direct Python C API from OPT #1 and OPT #2
Gaurav Sharma (bewithgaurav)
added a commit
that referenced
this pull request
Nov 10, 2025
Problem: - All numeric conversions used pybind11 wrappers with overhead: * Type detection, wrapper object creation, bounds checking * ~20-40 CPU cycles overhead per cell Solution: - Use direct Python C API calls: * PyLong_FromLong/PyLong_FromLongLong for integers * PyFloat_FromDouble for floats * PyBool_FromLong for booleans * PyList_SET_ITEM macro (no bounds check - list pre-sized) Changes: - SQL_INTEGER, SQL_SMALLINT, SQL_BIGINT, SQL_TINYINT → PyLong_* - SQL_BIT → PyBool_FromLong - SQL_REAL, SQL_DOUBLE, SQL_FLOAT → PyFloat_FromDouble - Added explicit NULL handling for each type Impact: - Eliminates pybind11 wrapper overhead for simple numeric types - Direct array access via PyList_SET_ITEM macro - Affects 7 common numeric SQL types
Gaurav Sharma (bewithgaurav)
added a commit
that referenced
this pull request
Nov 10, 2025
Gaurav Sharma (bewithgaurav)
added a commit
that referenced
this pull request
Nov 10, 2025
Problem: -------- Row creation and assignment had multiple layers of overhead: 1. Per-row allocation: py::list(numCols) creates pybind11 wrapper for each row 2. Cell assignment: row[col-1] = value uses pybind11 operator[] with bounds checking 3. Final assignment: rows[i] = row uses pybind11 list assignment with refcount overhead 4. Fragmented allocation: 1,000 separate py::list() calls instead of batch allocation For 1,000 rows: ~30-50 CPU cycles × 1,000 = 30K-50K wasted cycles Solution: --------- Replace pybind11 wrappers with direct Python C API throughout: 1. Row creation: PyList_New(numCols) instead of py::list(numCols) 2. Cell assignment: PyList_SET_ITEM(row, col-1, value) instead of row[col-1] = value 3. Final assignment: PyList_SET_ITEM(rows.ptr(), i, row) instead of rows[i] = row This completes the transition to direct Python C API started in OPT #2. Changes: -------- - Replaced py::list row(numCols) → PyObject* row = PyList_New(numCols) - Updated all NULL/SQL_NO_TOTAL handlers to use PyList_SET_ITEM - Updated all zero-length data handlers to use direct Python C API - Updated string handlers (SQL_CHAR, SQL_WCHAR) to use PyList_SET_ITEM - Updated complex type handlers (DECIMAL, DATETIME, DATE, TIME, TIMESTAMPOFFSET, GUID, BINARY) - Updated final row assignment to use PyList_SET_ITEM(rows.ptr(), i, row) All cell assignments now use direct Python C API: - Numeric types: Already done in OPT #2 (PyLong_FromLong, PyFloat_FromDouble, etc.) - Strings: PyUnicode_FromStringAndSize, PyUnicode_FromString - Binary: PyBytes_FromStringAndSize - Complex types: .release().ptr() to transfer ownership Impact: ------- - ✅ Eliminates pybind11 wrapper overhead for row creation - ✅ No bounds checking in hot loop (PyList_SET_ITEM is a macro) - ✅ Clean reference counting (objects created with refcount=1, transferred to list) - ✅ Consistent with OPT #2 (entire row/cell management via Python C API) - ✅ Expected 5-10% improvement (smaller than OPT #3, but completes the stack) All type handlers now bypass pybind11 for maximum performance.
Jahnvi Thakkar (jahnvi480)
added a commit
that referenced
this pull request
Sep 4, 2026
…e; aarch64 continueOnError Finding #1 (blocking): verify() ran `python -c import mssql_python` from the agent cwd (the checkout root, which holds the un-built mssql_python/ + mssql_python_odbc/ source). For python -c, sys.path[0] is '' (cwd), so the SOURCE shadowed the conda-installed package -> ImportError: No ddbc_bindings module found; the verify gate validated source (or failed) instead of the built package. Fix: a thin verify() wrapper os.chdir's to the neutral per-leg build dir before the phase (the Python equivalent of the deleted scripts' cd), so every verify + reachability subprocess inherits it; renamed the body to _verify_impl. New tests/test_034_conda_verify_cwd.py asserts the import probes run from the workdir (proven to FAIL without the wrapper) and is wired into the conda-audit gate. Finding #2 (medium): linux-aarch64 lacked continueOnError:true while sharing the BuildConda job with native linux-64 -- a QEMU flake failed the job and skipped the condition:succeeded() publish, discarding validated linux-64 packages. Added continueOnError:true (blast-radius isolation, matching win-arm64/osx-arm64; the release gate still requires the subdir) and reconciled the contradictory MANDATORY-blocking comment.
Jahnvi Thakkar (jahnvi480)
added a commit
that referenced
this pull request
Sep 4, 2026
…e; aarch64 continueOnError Finding #1 (blocking): verify() ran python -c import mssql_python from the agent cwd (checkout root with the un-built mssql_python/ + mssql_python_odbc/ source), so sys.path[0]='' shadowed the conda-installed package (ImportError: No ddbc_bindings module found). Fix: a thin verify() wrapper os.chdir's to the neutral per-leg build dir (Python equivalent of the deleted scripts' cd); body renamed to _verify_impl. New tests/test_034_conda_verify_cwd.py asserts the import probes run from the workdir (proven to fail without the wrapper). Finding #2 (medium): linux-aarch64 lacked continueOnError:true while sharing the BuildConda job with native linux-64 -- a QEMU flake failed the job and skipped the condition:succeeded() publish, discarding validated linux-64 packages. Added continueOnError:true (blast-radius isolation; release gate still requires the subdir) and reconciled the MANDATORY-blocking comment.
Jahnvi Thakkar (jahnvi480)
added a commit
that referenced
this pull request
Sep 7, 2026
…t odbc-wheel match PR #744 review #4: assert_macho_arch.py reads the Mach-O cputype slice(s) out of every vendored .dylib/.so and asserts the osx-64/osx-arm64 arch is present. osx-arm64 is cross-built on Intel where its runtime import is skipped, so this static pass is its only arch+presence check. Wired into audit_packages() next to the win-arm64 PE assert; test_035 mirrors test_030. PR #744 review #5: iter_payload_members now raises ValueError on an unrecognized extension (fail-closed like read_index) instead of silently yielding an empty iterator; test_029 covers it. PR #744 review #2: gather_wheels demands exactly one odbc-wheel-filter match instead of silently picking the first, so an arch-ambiguous filter cannot vendor the wrong-arch driver.
Jahnvi Thakkar (jahnvi480)
added a commit
that referenced
this pull request
Sep 7, 2026
…t odbc-wheel match PR #744 review #4: assert_macho_arch.py reads the Mach-O cputype slice(s) out of every vendored .dylib/.so and asserts the osx-64/osx-arm64 arch is present. osx-arm64 is cross-built on Intel where its runtime import is skipped, so this static pass is its only arch+presence check. Wired into audit_packages() next to the win-arm64 PE assert; test_035 mirrors test_030. PR #744 review #5: iter_payload_members now raises ValueError on an unrecognized extension (fail-closed like read_index) instead of silently yielding an empty iterator; test_029 covers it. PR #744 review #2: gather_wheels demands exactly one odbc-wheel-filter match instead of silently picking the first, so an arch-ambiguous filter cannot vendor the wrong-arch driver.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please accept this contribution adding the standard Microsoft SECURITY.MD 🔒 file to help the community understand the security policy and how to safely report security issues. GitHub uses the presence of this file to light-up security reminders and a link to the file. This pull request commits the latest official SECURITY.MD file from https://github.com/microsoft/repo-templates/blob/main/shared/SECURITY.md.
Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.