ArpeTeradata — Data types
How Teradata types map to Apache Arrow on the read path, and how Arrow types map back to Teradata for bound parameters and bulk ingest.
The non-obvious cases are called out below:
- Teradata has one binary floating type:
FLOAT/REAL/DOUBLE PRECISIONare allfloat64. TIMESTAMP WITH TIME ZONEis returned as the UTC instant;TIME WITH TIME ZONEandINTERVALcome back as text.- An unbounded
NUMBERcomes back as lossless decimal text. PERIODreads as a struct,ARRAYandVECTORas lists.- The
GRAPHICfamily cannot be read under aUTF8session; translate it on the server.
Session settings that affect types
- The session character set is
UTF8(adbc.arpeteradata.charset, the only value accepted). DECIMALprecision up to 38 is requested at logon. Without it, Teradata would capDECIMALat 18.- The session time zone is set to UTC right after logon, whatever the
site, user or profile default.
CURRENT_TIMESTAMPand other session-time functions therefore run in UTC, and bound time-zone-aware Arrow timestamps land unchanged. ASET TIME ZONEsent by your application changes this for the rest of the session. - Times and timestamps carry microsecond precision, Teradata's maximum. A column declared with fewer fractional digits is widened.
Read path: Teradata → Arrow
Numeric
| Teradata type | Arrow type | Notes |
|---|---|---|
BYTEINT | int8 | |
SMALLINT | int16 | |
INTEGER | int32 | |
BIGINT | int64 | |
FLOAT, REAL, DOUBLE PRECISION | float64 | Teradata has one binary floating type. |
DECIMAL(p,s) / NUMERIC(p,s), p 1–38 | decimal128(p,s) | Exact. |
NUMBER(p,s), p 1–38 | decimal128(p,s) | Each value is rescaled to the column scale. |
NUMBER with no precision | utf8 | Lossless decimal text without an exponent: 1.5E100 comes back as 15 followed by 99 zeros. |
Character and binary
| Teradata type | Arrow type | Notes |
|---|---|---|
CHAR(n) | utf8 | Trailing blanks are trimmed. |
VARCHAR(n), LONG VARCHAR | utf8 | |
BYTE(n) | binary | |
VARBYTE(n), LONG VARBYTE | binary | |
GRAPHIC, VARGRAPHIC, LONG VARGRAPHIC | — | Refused by the server under a UTF8 session; see Unsupported types. |
Temporal
| Teradata type | Arrow type | Notes |
|---|---|---|
DATE | date32 | Dates before 1900 are handled (0001-01-01 round-trips). |
TIME(n) | time64[us] | |
TIMESTAMP(n) | timestamp[us] (no time zone) | |
TIMESTAMP(n) WITH TIME ZONE | timestamp[us, UTC] | The driver applies the stored offset and returns the UTC instant. |
TIME(n) WITH TIME ZONE | utf8 | Arrow has no time-zone-aware time type. The text keeps its offset, e.g. 12:34:56.123456+05:30. |
INTERVAL (all 13 forms, YEAR to DAY TO SECOND) | utf8 | Teradata's fixed-width text form, including the leading sign column, e.g. 12 03:04:05.123456. |
PERIOD, user-defined types, ARRAY, VECTOR
| Teradata type | Arrow type | Notes |
|---|---|---|
PERIOD(DATE) | struct<begin: date32, end: date32> | A NULL period is a null struct. |
PERIOD(TIME(n)) | struct<begin: time64[us], end: time64[us]> | |
PERIOD(TIMESTAMP(n)) | struct<begin: timestamp[us], end: timestamp[us]> | |
PERIOD(TIMESTAMP(n) WITH TIME ZONE) | struct<begin: timestamp[us, UTC], end: timestamp[us, UTC]> | UTC instants, as for the scalar type. |
PERIOD(TIME(n) WITH TIME ZONE) | struct<begin: utf8, end: utf8> | Text with offset, as for the scalar type. |
| Distinct user-defined type | the base type's mapping | CREATE TYPE euro AS DECIMAL(10,2) reads as decimal128(10,2). |
ARRAY / VARRAY, one dimension | list<element> | Elements of type BYTEINT, SMALLINT, INTEGER, BIGINT, FLOAT, DECIMAL / NUMBER(p,s), CHAR, VARCHAR, DATE, TIME, TIMESTAMP take their scalar mapping; NULL elements are null items. Other element types come back as the server's text, e.g. (1,2,3). |
ARRAY, n dimensions | n nested lists | INTEGER ARRAY[2][3] reads as [[1,2,3],[4,5,6]]. |
VECTOR | list<float64> | One list per row; rows can differ in length. |
VECTOR32 | list<float32> | |
Structured or internal user-defined type (including MBR) | utf8 | The type's transform text, e.g. (0,0,1,1). |
Teradata sends user-defined values through their transform, which for an
ARRAY is text: FLOAT elements therefore carry the 15 significant digits the
server writes. The first time a connection reads an ARRAY or structured type,
it looks the type up once and caches it for the connection's lifetime. A
statement is never executed twice for this. On that first read, a
user-defined column returned by a CALL or a multi-statement request can
come back as the server's text.
LOBs and documents
| Teradata type | Arrow type | Notes |
|---|---|---|
CLOB (LATIN or UNICODE) | large_string | UTF-8 text. |
XML | large_string | The document text, as the server stores it. |
DATASET STORAGE FORMAT CSV | large_string | |
JSON (text, BSON, UBJSON, LATIN) | large_string, extension arrow.json | JSON text whatever the storage format. |
BLOB | large_binary | |
DATASET STORAGE FORMAT AVRO | large_binary | Verified on NULL values and the column type only. |
ST_Geometry | large_string, extension geoarrow.wkt | WKT text such as LINESTRING (0 0,1 1). Teradata's WKT carries no SRID, so no CRS is named. |
Extensions are carried in the Arrow field metadata
(ARROW:extension:name, ARROW:extension:metadata): a consumer that does not
know them sees a plain large_string. An empty LOB and a NULL LOB are
distinct. A single LOB value up to Teradata's 2 GB limit reads.
Rows over 64 KB
Rows up to Teradata's large-row limit (about 1 MB) read. One limit is the
server's: with a UTF8 session, Teradata exports at most 64000 bytes of a
VARCHAR value, so a 32000-character CJK value comes back as 21333 characters
(Teradata's own drivers behave the same).
Type names in field metadata
Every result field carries an ARPETERADATA:type key in its Arrow field
metadata: the column's Teradata type, spelled as Teradata DDL spells it. It is
present on query results, on ExecuteSchema and on GetTableSchema, so
Teradata types that share an Arrow type (utf8 for VARCHAR, CHAR,
TIME WITH TIME ZONE and INTERVAL, for example) stay distinguishable.
Since v0.2.9.
| Column | ARPETERADATA:type |
|---|---|
VARCHAR(100) CHARACTER SET UNICODE | VARCHAR(100) |
DECIMAL(10,2) | DECIMAL(10,2) |
NUMBER(*,2) | NUMBER(*,2) |
TIMESTAMP(6) WITH TIME ZONE | TIMESTAMP(6) WITH TIME ZONE |
INTERVAL DAY(4) TO SECOND(2) | INTERVAL DAY(4) TO SECOND(2) |
PERIOD(DATE) | PERIOD(DATE) |
ST_Geometry and other user-defined types | SYSUDTLIB.ST_Geometry |
Character lengths are in characters, and the character set is not part of
the name. User-defined types, including ST_Geometry, VECTOR and ARRAY
types, are named SYSUDTLIB.<name>.
Unsupported types
Unsupported types fail loudly: the driver never returns wrong or empty values for them.
GRAPHIC, VARGRAPHIC, LONG VARGRAPHIC are refused by the server under
the UTF8 session character set (Teradata errors 6705 and 6701, reported as
INVALID_ARGUMENT). Teradata exports GRAPHIC only to a KANJI session
character set, and Teradata's own drivers fail the same way. Convert the
column on the server:
SELECT TRANSLATE(vg USING GRAPHIC_TO_UNICODE) AS vg FROM t;
The result is ordinary Unicode text (utf8). A fixed GRAPHIC(n) value keeps
its padding of ideographic spaces (U+3000).
Any other type that reaches the driver without a mapping fails when the
schema is built, naming the type code and the column:
unsupported column type code <n> for column <name>.
Bound parameters: Arrow → Teradata
Values bound to ? markers (Bind, Python's executemany) accept these
Arrow types:
| Arrow type | Sent as |
|---|---|
int8, int16, int32, int64 | BYTEINT, SMALLINT, INTEGER, BIGINT |
float64 | FLOAT |
decimal32 / decimal64 / decimal128 / decimal256 (p,s), p 1–38 | DECIMAL(p,s) |
utf8 | VARCHAR, at most 64000 UTF-8 bytes per value |
binary | VARBYTE |
date32 | DATE |
time64[us] | TIME(n), with n the fewest fractional digits the bound values need |
timestamp[us] without a time zone | TIMESTAMP(n), n as for TIME |
timestamp[us] with a time zone | TIMESTAMP(n) WITH TIME ZONE at +00:00: a WITH TIME ZONE column stores the same instant, a plain TIMESTAMP column stores it in UTC |
| null type | NULL |
Also accepted, for bound parameters and for ingest (since v0.3.2): bool,
uint8 / uint16 / uint32 / uint64, float32, large_string,
large_binary, fixed_size_binary, date64, time32[s|ms], time64[ns]
and timestamp[s|ms|ns]. Nanoseconds are truncated to microseconds.
Every other Arrow type is refused before anything is sent, including decimals
with a precision over 38 and dictionary-encoded columns. Dates and timestamps
outside years 1 to 9999, and times of 24 hours or more, are refused with
value out of range. A string or binary value over 64000 bytes is refused.
Because the declared precision follows the data, whole-second values bind into
TIME(0) and TIMESTAMP(0) columns; a value with a fraction is refused by
Teradata for such a column (error 5404) rather than truncated.
Ingest: Arrow → Teradata column types
When adbc_ingest creates the table (create, replace, create_append),
the Arrow schema gives the column types. Since v0.3.2.
| Arrow type | Teradata column |
|---|---|
bool | BYTEINT (0 or 1) |
int8, int16, int32, int64 | BYTEINT, SMALLINT, INTEGER, BIGINT |
uint8, uint16, uint32 | SMALLINT, INTEGER, BIGINT |
uint64 | DECIMAL(20,0) |
float32, float64 | FLOAT |
decimal32 / decimal64 / decimal128 / decimal256 (p,s), p up to 38 | DECIMAL(p,s) |
utf8, large_utf8 | VARCHAR(n) CHARACTER SET UNICODE, n = adbc.arpeteradata.ingest.varchar_length (default 4000) |
binary, large_binary | VARBYTE(2n), capped at 64000 |
fixed_size_binary(n) | BYTE(n) |
date32, date64 | DATE |
time32[s], time32[ms], time64[us], time64[ns] | TIME(0), TIME(3), TIME(6), TIME(6) |
timestamp[s] / [ms] / [us] / [ns] without a time zone | TIMESTAMP(0), TIMESTAMP(3), TIMESTAMP(6), TIMESTAMP(6) |
timestamp with a time zone | the same, WITH TIME ZONE |
| null type | VARCHAR(1) CHARACTER SET UNICODE |
- A non-nullable Arrow field becomes a
NOT NULLcolumn. - Created tables are
MULTISETandNO PRIMARY INDEX. To get a primary index, create the table yourself and ingest withappend. - Nanosecond times and timestamps are truncated to microseconds: the one lossy case.
- String width. A string longer than its column is truncated silently
in Teradata (BTET) session mode, and refused in ANSI mode (error 3996). This
also applies to
appendinto a narrower existing column. Raiseadbc.arpeteradata.ingest.varchar_length(up to 32000), or create the table yourself, wide enough, and useappend. - A table whose declared row is too wide is refused by Teradata at
CREATE(error 3933): lowervarchar_length. string_view,binary_view, dictionary, list, struct, map, interval and duration columns fail withNOT_IMPLEMENTED, before anything is inserted.- With
append, the driver creates no columns: Teradata converts the values to the existing column types.
See also
- Connection: the ingest options
- Compatibility: ingest behaviour and transactions
- Examples: bind and ingest recipes