Skip to content

Commit ace8924

Browse files
llaliEma PaunovicMaksim Vlasov
authored
Sync repos: Release 180.37.3 (#214)
* Merged PR 2076477: adding CREATE SEMANTIC INDEX feature Syntax CREATE SEMANTIC INDEX index_name ON table_name (column_definition [,...n] ) WITH (index_option [, ...n]) [ON {FILEGROUP <filegroup_name> | DEFAULT}] [;] column_definition :: = column_name [SEARCH_TYPE {vector | fulltext | hybrid}] [TYPE COLUMN type_column_name] [LANGUAGE language_term] [CHUNK_USING(chunk_option [,...n] )] chunk_option ::= { TYPE = {fixed | sentence | paragraph chapter | SIZE = character_count_per_chunk | OVERLAP = character_overlap_percentage } index_option ::= { EXTERNAL_MODEL = model_name [ (model_option) ] [VECTOR_INDEX ( vector_index_option [,...n] )] [FULLTEXT_STOPLIST = {OFF | SYSTEM | stoplist_name}] [MAXDOP = degree_of_parallelism] [DROP_EXISTING = { OFF | ON }] } model_option ::= { PARAMETERS = json_config } vector_index_option ::= { METRIC = {'cosine' | 'dot' | 'euclidean'} | TYPE = 'diskann' } - [X] The [Common checklist](https://msdata.visualstudio.com/SQLToolsAndLibraries/_git/Common?path=/Templates/PR%20Checklist%20for%20SQLToolsAndLibraries.md&version=GBmain&_a=preview) has been reviewed and followed - [X] Code changes are accompanied by appropriate unit tests - [ ] Identified and included SMEs needed to review code changes - [ ] Follow the [steps](https://msdata.visualstudio.com/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki/33838/Adding-or-Extending-TSql-support-in-Sql-Dom?anchor=make-the-changes-in) here to make changes in the code - [X] Follow the [steps](https://msdata.visualstudio.com/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki/33838/Adding-or-Extending-TSql-support-in-Sql-Dom?anchor=to-extend-the-tests-do-the-following%3A) here to add new tests for your feature - [ ] Update relevant documentation in the [wiki](https://dev.azure.com/msdata/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki) and the README.md file Please provide any additional information that might be helpful for the reviewers adding create semanic index feature ---- New feature implementation adding support for CREATE SEMANTIC INDEX statement in SQL Server's T-SQL parser (TSql180). This PR implements comprehensive parsing and script generation support for the new CREATE SEMANTIC INDEX statement, which enables semantic indexing with vector search, full-text search, and hybrid search capabilities. The feature includes column-level options for search types, chunk... * Merged PR 2111576: [Trident DW] Support invoke_external_api intrinsic and its' UDF syntax # Pull Request Template for ScriptDom ## Description This PR adds support for a new intrinsic that we've been implementing in Fabric DataWarehouse: `invoke_external_api` and its' UDF syntax sugar which later must become the only supported syntax - `EXTERNAL FUNCTION` ## Code Change - [x] The [Common checklist](https://msdata.visualstudio.com/SQLToolsAndLibraries/_git/Common?path=/Templates/PR%20Checklist%20for%20SQLToolsAndLibraries.md&version=GBmain&_a=preview) has been reviewed and followed - [x] Code changes are accompanied by appropriate unit tests - [x] Identified and included SMEs needed to review code changes - [x] Follow the [steps](https://msdata.visualstudio.com/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki/33838/Adding-or-Extending-TSql-support-in-Sql-Dom?anchor=make-the-changes-in) here to make changes in the code ## Testing - [x] Follow the [steps](https://msdata.visualstudio.com/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki/33838/Adding-or-Extending-TSql-support-in-Sql-Dom?anchor=to-extend-the-tests-do-the-following%3A) here to add new tests for your feature ## Documentation - [x] Update relevant documentation in the [wiki](https://dev.azure.com/msdata/SQLToolsAndLibraries/_wiki/wikis/SQLToolsAndLibraries.wiki) and the README.md file - no changes requried ## Additional Information [AI and Extensibility.docx](https://microsoft.sharepoint.com/:w:/r/teams/AzurePolaris/_layouts/15/doc2.aspx?sourcedoc=%7B75BAC667-A870-4482-8A37-F80E6EC8FCE0%7D&file=AI%20and%20Extensibiliy.docx) `VARCHAR(MAX) invoke_external_api('functionSetName', 'functionName'[, ... args])` - invokes Fabric Function `functionName` in `functionSetName` where Function Set is `User Defined Functions` item in PowerBI Workspace. `CREATEA FUNCTION [schema].[localName] [RETURNS <type>] AS EXTERNAL FUNCTION functionSetName.functionName` is a DDL syntax used to create SQL UDF based on function from Function Set. The DML would be: `SELECT [schema].[localName](col1) FROM tbl`, which during expression tree transformation will become `invoke_external_api` ---- #### AI description (iteration 3) #### PR Classification New feature: Adding support for external function definitions and the `INVOKE_EXTERNAL_API` intrinsic function in Trident Data Warehouse (Fabric DW). #### PR Summary This PR introduces syntax support for creating external functions and invoking external APIs in SQL ScriptDom for Fabric DW. It enables defining function aliases to external function sets and calling them via a new intrinsic. - **`TSqlFabricDW.g`**: Added grammar rules for `CREATE/ALTER/CREATE OR ALTER FUNCTION ... AS EXTERNAL FUNCTION` statements and `INVOKE_EXTERNAL_API` function call syntax - **`Ast.xml`**: Defined new AST node types `ExternalFunctionStatement` (and variants) and `InvokeExternalApiFunctionCall` with their properties - **`SqlScriptGeneratorVisitor.ExternalFunctionStatement.cs`** and **`SqlScriptGeneratorVisitor.InvokeExternalApiFunction.cs`**: Implemen... * Merged PR 2150081: Adding release notes for 180.37.3 adding release notes for 180.37.3 ---- #### AI description (iteration 2) #### PR Classification This is a documentation update adding release notes for version 180.35.0 of Microsoft.SqlServer.TransactSql.ScriptDom. #### PR Summary This pull request adds comprehensive release notes documenting new features, bug fixes, and improvements in version 180.35.0 of the SQL Script DOM library. - `/release-notes/180/180.37.3.md`: Documents support for CREATE FUNCTION AS EXTERNAL FUNCTION in Microsoft Fabric Warehouse - `/release-notes/180/180.37.3.md`: Lists bug fixes for PERSIST_SAMPLE_PERCENT parsing, AlterTableAddTableElementStatement separator issues, and WindowDefinition whitespace handling - `/release-notes/180/180.37.3.md`: Notes improvements to comment preservation in SQL Server script generator and .NET SDK update to version 8.0.420 <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> --------- Co-authored-by: Ema Paunovic <emapaunovic@microsoft.com> Co-authored-by: Maksim Vlasov <maksimvlasov@microsoft.com>
1 parent 20f9bf1 commit ace8924

20 files changed

Lines changed: 1258 additions & 3 deletions

‎SqlScriptDom/Parser/TSql/Ast.xml‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -794,6 +794,30 @@
794794
<InheritedMember Name="MethodSpecifier" ContainerClass ="ProcedureStatementBodyBase" />
795795
<InheritedMember Name="OrderHint" ContainerClass ="FunctionStatementBody" />
796796
</Class>
797+
<Class Name="ExternalFunctionStatement" Base="TSqlStatement" Abstract="true" Summary="Represents the CREATE/ALTER FUNCTION ... AS EXTERNAL FUNCTION statement.">
798+
<Member Name="Name" Type="SchemaObjectName" Summary="The name of the function being created or altered. Can at most be a two part name."/>
799+
<Member Name="Parameters" Type="ProcedureParameter" Collection="true" Summary="Parameters which can be passed to the external function."/>
800+
<Member Name="ReturnType" Type="DataTypeReference" Summary="Optional RETURNS clause specifying the data type returned by the external function."/>
801+
<Member Name="ExternalName" Type="SchemaObjectName" Summary="The external function reference of the form functionSetName.functionName."/>
802+
</Class>
803+
<Class Name="CreateExternalFunctionStatement" Base="ExternalFunctionStatement" Summary="Represents the CREATE FUNCTION ... AS EXTERNAL FUNCTION statement.">
804+
<InheritedMember Name="Name" ContainerClass ="ExternalFunctionStatement" />
805+
<InheritedMember Name="Parameters" ContainerClass ="ExternalFunctionStatement" />
806+
<InheritedMember Name="ReturnType" ContainerClass ="ExternalFunctionStatement" />
807+
<InheritedMember Name="ExternalName" ContainerClass ="ExternalFunctionStatement" />
808+
</Class>
809+
<Class Name="AlterExternalFunctionStatement" Base="ExternalFunctionStatement" Summary="Represents the ALTER FUNCTION ... AS EXTERNAL FUNCTION statement.">
810+
<InheritedMember Name="Name" ContainerClass ="ExternalFunctionStatement" />
811+
<InheritedMember Name="Parameters" ContainerClass ="ExternalFunctionStatement" />
812+
<InheritedMember Name="ReturnType" ContainerClass ="ExternalFunctionStatement" />
813+
<InheritedMember Name="ExternalName" ContainerClass ="ExternalFunctionStatement" />
814+
</Class>
815+
<Class Name="CreateOrAlterExternalFunctionStatement" Base="ExternalFunctionStatement" Summary="Represents the CREATE OR ALTER FUNCTION ... AS EXTERNAL FUNCTION statement.">
816+
<InheritedMember Name="Name" ContainerClass ="ExternalFunctionStatement" />
817+
<InheritedMember Name="Parameters" ContainerClass ="ExternalFunctionStatement" />
818+
<InheritedMember Name="ReturnType" ContainerClass ="ExternalFunctionStatement" />
819+
<InheritedMember Name="ExternalName" ContainerClass ="ExternalFunctionStatement" />
820+
</Class>
797821
<Class Name="CreateRuleStatement" Base="TSqlStatement" Summary="Represents the Create Rule statement.">
798822
<Member Name="Name" Type="SchemaObjectName" Summary="Represents the name. DatabaseName is disallowed here, has to equal String.Empty or null."/>
799823
<Member Name="Expression" Type="BooleanExpression" Summary="Expression that is the rule."/>
@@ -4658,6 +4682,28 @@
46584682
<InheritedMember Name="IndexOptions" ContainerClass="IndexStatement" />
46594683
<Member Name="OnFileGroupOrPartitionScheme" Type="FileGroupOrPartitionScheme" Summary="The filegroup or partition scheme. Might be null."/>
46604684
</Class>
4685+
<Class Name="SemanticIndexChunkOption" Summary="Represents a chunk option in a CHUNK_USING clause of a semantic index column.">
4686+
<Member Name="OptionKind" Type="SemanticIndexChunkOptionKind" GenerateUpdatePositionInfoCall="false" Summary="The kind of chunk option (Type, Size, or Overlap)."/>
4687+
<Member Name="Value" Type="IdentifierOrValueExpression" Summary="The value of the chunk option."/>
4688+
</Class>
4689+
<Class Name="SemanticIndexColumn" Summary="Represents a column definition in a CREATE SEMANTIC INDEX statement.">
4690+
<Member Name="ColumnName" Type="Identifier" Summary="The column name."/>
4691+
<Member Name="SearchType" Type="SemanticIndexSearchType" GenerateUpdatePositionInfoCall="false" Summary="The search type (vector, fulltext, hybrid). NotSpecified if not specified."/>
4692+
<Member Name="TypeColumnName" Type="Identifier" Summary="The TYPE COLUMN name. Optional may be null."/>
4693+
<Member Name="Language" Type="IdentifierOrValueExpression" Summary="The LANGUAGE term. Optional may be null."/>
4694+
<Member Name="ChunkOptions" Type="SemanticIndexChunkOption" Collection="true" Summary="The CHUNK_USING options. Optional may have zero elements."/>
4695+
</Class>
4696+
<Class Name="CreateSemanticIndexStatement" Base="IndexStatement" Summary="Represents the CREATE SEMANTIC INDEX statement.">
4697+
<InheritedMember Name="Name" ContainerClass="IndexStatement" />
4698+
<InheritedMember Name="OnName" ContainerClass="IndexStatement" />
4699+
<Member Name="Columns" Type="SemanticIndexColumn" Collection="true" Summary="The column definitions."/>
4700+
<Member Name="ExternalModelName" Type="Identifier" Summary="The external model name. Optional may be null."/>
4701+
<Member Name="ExternalModelParameters" Type="StringLiteral" Summary="The PARAMETERS value for the external model. Optional may be null."/>
4702+
<Member Name="VectorIndexOptions" Type="IndexOption" Collection="true" Summary="The VECTOR_INDEX options (METRIC, TYPE). Optional may have zero elements."/>
4703+
<Member Name="FulltextStoplistOption" Type="StopListFullTextIndexOption" Summary="The FULLTEXT_STOPLIST option. Optional may be null."/>
4704+
<InheritedMember Name="IndexOptions" ContainerClass="IndexStatement" />
4705+
<Member Name="OnFileGroupOrPartitionScheme" Type="FileGroupOrPartitionScheme" Summary="The filegroup or partition scheme. Might be null."/>
4706+
</Class>
46614707
<Class Name="WindowFrameClause" Summary="Represents the specification of window bounds for windowing aggregates.">
46624708
<Member Name="Top" Type="WindowDelimiter" Summary="Top boundary of the window."/>
46634709
<Member Name="Bottom" Type="WindowDelimiter" Summary="Bottom boundary of the window. Optional may be null."/>
@@ -4806,6 +4852,12 @@
48064852
<Member Name="Input" Type="ScalarExpression" Summary="Text to translate; variable or column reference." />
48074853
<Member Name="Language" Type="ScalarExpression" Summary="Target language; typically string literal or identifier." />
48084854
</Class>
4855+
<Class Name="InvokeExternalApiFunctionCall" Base="PrimaryExpression" Summary="Represents INVOKE_EXTERNAL_API(...) function call.">
4856+
<InheritedClass Name="PrimaryExpression" />
4857+
<Member Name="FunctionSetName" Type="StringLiteral" Summary="Function set name (string literal, required)." />
4858+
<Member Name="FunctionName" Type="StringLiteral" Summary="Function name (string literal, required)." />
4859+
<Member Name="Arguments" Type="ScalarExpression" Collection="true" Summary="Optional additional arguments passed to the external API." />
4860+
</Class>
48094861
<Class Name="VectorSearchTableReference" Base="TableReferenceWithAlias" Summary="Represents the VECTOR_SEARCH table-valued function call.">
48104862
<InheritedClass Name="TableReferenceWithAlias" />
48114863
<Member Name="Table" Type="TableReferenceWithAlias" Summary="Table on which perform the search." />

‎SqlScriptDom/Parser/TSql/CodeGenerationSupporter.cs‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -204,6 +204,7 @@ internal static class CodeGenerationSupporter
204204
internal const string ChangeTable = "CHANGETABLE";
205205
internal const string ChangeTracking = "CHANGE_TRACKING";
206206
internal const string ChangeTrackingContext = "CHANGE_TRACKING_CONTEXT";
207+
internal const string Chapter = "CHAPTER";
207208
internal const string Char = "CHAR";
208209
internal const string CharacterSet = "CHARACTER_SET";
209210
internal const string CheckConstraints = "CHECK_CONSTRAINTS";
@@ -214,6 +215,7 @@ internal static class CodeGenerationSupporter
214215
internal const string ChecksumAgg = "CHECKSUM_AGG";
215216
internal const string ChunkSize = "CHUNK_SIZE";
216217
internal const string ChunkType = "CHUNK_TYPE";
218+
internal const string ChunkUsing = "CHUNK_USING";
217219
internal const string ModularSum = "MODULAR_SUM";
218220
internal const string Classifier = "CLASSIFIER";
219221
internal const string Classification = "CLASSIFICATION";
@@ -394,6 +396,7 @@ internal static class CodeGenerationSupporter
394396
internal const string Extension = "EXTENSION";
395397
internal const string External = "EXTERNAL";
396398
internal const string ExternalAccess = "EXTERNAL_ACCESS";
399+
internal const string ExternalModel = "EXTERNAL_MODEL";
397400
internal const string ExternalMonitor = "EXTERNAL_MONITOR";
398401
internal const string Extract = "EXTRACT";
399402
internal const string FailOperation = "FAIL_OPERATION";
@@ -465,6 +468,7 @@ internal static class CodeGenerationSupporter
465468
internal const string Full = "FULL";
466469
internal const string FullScan = "FULLSCAN";
467470
internal const string Fulltext = "FULLTEXT";
471+
internal const string FulltextStopList = "FULLTEXT_STOPLIST";
468472
internal const string General = "GENERAL";
469473
internal const string GenerateSeries = "GENERATE_SERIES";
470474
internal const string Generated = "GENERATED";
@@ -499,6 +503,7 @@ internal static class CodeGenerationSupporter
499503
internal const string Hidden = "HIDDEN";
500504
internal const string High = "HIGH";
501505
internal const string Hint = "HINT";
506+
internal const string Hybrid = "HYBRID";
502507
internal const string Histogram = "HISTOGRAM";
503508
internal const string HistogramSteps = "HISTOGRAM_STEPS";
504509
internal const string HistoryRetentionPeriod = "HISTORY_RETENTION_PERIOD";
@@ -536,6 +541,7 @@ internal static class CodeGenerationSupporter
536541
internal const string Intermediate = "INTERMEDIATE";
537542
internal const string IntervalLengthMinutes = "INTERVAL_LENGTH_MINUTES";
538543
internal const string Insensitive = "INSENSITIVE";
544+
internal const string InvokeExternalApi = "INVOKE_EXTERNAL_API";
539545
internal const string IRowset = "IROWSET";
540546
internal const string Isolation = "ISOLATION";
541547
internal const string Job = "JOB";
@@ -755,6 +761,7 @@ internal static class CodeGenerationSupporter
755761
internal const string PageCount = "PAGECOUNT";
756762
internal const string PageVerify = "PAGE_VERIFY";
757763
internal const string PagLock = "PAGLOCK";
764+
internal const string Paragraph = "PARAGRAPH";
758765
internal const string Param = "PARAM";
759766
internal const string Parameter = "PARAMETER";
760767
internal const string Parameters = "PARAMETERS";
@@ -926,6 +933,7 @@ internal static class CodeGenerationSupporter
926933
internal const string Scroll = "SCROLL";
927934
internal const string ScrollLocks = "SCROLL_LOCKS";
928935
internal const string Search = "SEARCH";
936+
internal const string SearchType = "SEARCH_TYPE";
929937
internal const string Secondary = "SECONDARY";
930938
internal const string SecondaryRole = "SECONDARY_ROLE";
931939
internal const string Seconds = "SECONDS";
@@ -935,6 +943,8 @@ internal static class CodeGenerationSupporter
935943
internal const string SecurityLog = "SECURITY_LOG";
936944
internal const string Selective = "SELECTIVE";
937945
internal const string Self = "SELF";
946+
internal const string Semantic = "SEMANTIC";
947+
internal const string Sentence = "SENTENCE";
938948
internal const string SemiColon = ";";
939949
internal const string Send = "SEND";
940950
internal const string Sensitivity = "SENSITIVITY";
@@ -1123,6 +1133,7 @@ internal static class CodeGenerationSupporter
11231133
internal const string Varp = "VARP";
11241134
internal const string VDevNo = "VDEVNO";
11251135
internal const string Vector = "Vector";
1136+
internal const string VectorIndex = "VECTOR_INDEX";
11261137
internal const string VectorSearch = "VECTOR_SEARCH";
11271138
internal const string Verbose = "VERBOSE";
11281139
internal const string VerboseLogging = "VerboseLogging";

‎SqlScriptDom/Parser/TSql/SemanticIndexChunkOptionKind.cs‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
//------------------------------------------------------------------------------
2+
// <copyright file="SemanticIndexChunkOptionKind.cs" company="Microsoft">
3+
// Copyright (c) Microsoft Corporation. All rights reserved.
4+
// </copyright>
5+
//------------------------------------------------------------------------------
6+
7+
8+
9+
namespace Microsoft.SqlServer.TransactSql.ScriptDom
10+
{
11+
#pragma warning disable 1591
12+
13+
/// <summary>
14+
/// The possible values for semantic index chunk options.
15+
/// </summary>
16+
public enum SemanticIndexChunkOptionKind
17+
{
18+
Type = 0,
19+
Size = 1,
20+
Overlap = 2
21+
}
22+
23+
#pragma warning restore 1591
24+
}

‎SqlScriptDom/Parser/TSql/SemanticIndexSearchType.cs‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
//------------------------------------------------------------------------------
2+
// <copyright file="SemanticIndexSearchType.cs" company="Microsoft">
3+
// Copyright (c) Microsoft Corporation. All rights reserved.
4+
// </copyright>
5+
//------------------------------------------------------------------------------
6+
7+
8+
9+
namespace Microsoft.SqlServer.TransactSql.ScriptDom
10+
{
11+
#pragma warning disable 1591
12+
13+
/// <summary>
14+
/// The possible values for semantic index search type.
15+
/// </summary>
16+
public enum SemanticIndexSearchType
17+
{
18+
NotSpecified = 0,
19+
Vector = 1,
20+
Fulltext = 2,
21+
Hybrid = 3
22+
}
23+
24+
#pragma warning restore 1591
25+
}

0 commit comments

Comments
 (0)