Skip to main content
Version: Main branch

datatype

superduper.components.datatype

Source code

base64_to_bytes​

base64_to_bytes(encoded)
ParameterDescription
encodedThe base64 encoded string.

Decodes a base64 encoded string.

bytes_to_base64​

bytes_to_base64(bytes)
ParameterDescription
bytesThe bytes to convert.

Converts bytes to base64.

dill_decode​

dill_decode(b: bytes,
info: Optional[Dict] = None) -> Any
ParameterDescription
bThe bytes to decode.
infoOptional information.

Decodes bytes using dill.

dill_encode​

dill_encode(object: Any,
info: Optional[Dict] = None) -> bytes
ParameterDescription
objectThe object to encode.
infoOptional information.

Encodes an object using dill.

encode_torch_state_dict​

encode_torch_state_dict(module,
info)
ParameterDescription
moduleModule.
infoInformation.

Encode torch state dictionary.

file_check​

file_check(path: Any,
info: Optional[Dict] = None) -> str
ParameterDescription
pathThe file path to check.
infoOptional information.

Checks if a file path exists.

get_serializer​

get_serializer(identifier: str,
method: str,
encodable: str = 'encodable',
db: Optional[ForwardRef('Datalayer')] = None)
ParameterDescription
identifierThe identifier of the serializer.
methodThe method of the serializer.
encodableThe type of encodable object.
dbThe Datalayer instance.

Get a serializer.

json_decode​

json_decode(b: str,
info: Optional[Dict] = None) -> Any
ParameterDescription
bThe JSON string to decode
infoOptional information

Decode the JSON string to an dict.

json_encode​

json_encode(object: Any,
info: Optional[Dict] = None) -> str
ParameterDescription
objectThe object to encode
infoOptional information

Encode the dict to a JSON string.

pickle_decode​

pickle_decode(b: bytes,
info: Optional[Dict] = None) -> Any
ParameterDescription
bThe bytes to decode.
infoOptional information.

Decodes bytes using pickle.

pickle_encode​

pickle_encode(object: Any,
info: Optional[Dict] = None) -> bytes
ParameterDescription
objectThe object to encode.
infoOptional information.

Encodes an object using pickle.

torch_decode​

torch_decode(b: bytes,
info: Optional[Dict] = None) -> Any
ParameterDescription
bThe bytes to decode.
infoOptional information.

Decodes bytes to a torch model.

torch_encode​

torch_encode(object: Any,
info: Optional[Dict] = None) -> bytes
ParameterDescription
objectThe object to encode.
infoOptional information.

Saves an object in torch format.

DataType​

DataType(self,
identifier: str,
db: dataclasses.InitVar[typing.Optional[ForwardRef('Datalayer')]] = None,
uuid: None = <factory>,
*,
upstream: "t.Optional[t.List['Component']]" = None,
plugins: "t.Optional[t.List['Plugin']]" = None,
artifacts: 'dc.InitVar[t.Optional[t.Dict]]' = None,
cache: bool = True,
status: 't.Optional[Status]' = None,
encoder: Optional[Callable] = None,
decoder: Optional[Callable] = None,
info: Optional[Dict] = None,
shape: Optional[Sequence] = None,
directory: Optional[str] = None,
encodable: str = 'encodable',
bytes_encoding: Optional[str] = <BytesEncoding.BYTES: 'bytes'>,
intermediate_type: Optional[str] = 'bytes',
media_type: Optional[str] = None) -> None
ParameterDescription
identifierIdentifier of the leaf.
dbDatalayer instance.
uuidUUID of the leaf.
artifactsA dictionary of artifacts paths and DataType objects
upstreamA list of upstream components
pluginsA list of plugins to be used in the component.
cache(Optional) If set true the component will not be cached during primary job of the component i.e on a distributed cluster this component will be reloaded on every component task e.g model prediction.
statusWhat part of the lifecycle the component is in.
encoderA callable that converts an encodable object of this encoder to bytes.
decoderA callable that converts bytes to an encodable object of this encoder.
infoAn optional information dictionary.
shapeThe shape of the data.
directoryThe directory to store file types.
encodableThe type of encodable object ('encodable', 'lazy_artifact', or 'file').
bytes_encodingThe encoding type for bytes ('base64' or 'bytes').
intermediate_typeType of the intermediate data [IntermediateType.BYTES, IntermediateType.STRING]
media_typeThe media type.

A data type component that defines how data is encoded and decoded.

Artifact​

Artifact(self,
db: dataclasses.InitVar[typing.Optional[ForwardRef('Datalayer')]] = None,
uuid: None = <factory>,
*,
identifier: str = '',
datatype: superduper.components.datatype.DataType,
uri: Optional[str] = None,
x: Any = <EMPTY>,
blob: dataclasses.InitVar[typing.Union[str,
bytes,
NoneType]] = None) -> None
ParameterDescription
identifierIdentifier of the leaf.
dbDatalayer instance.
uuidUUID of the leaf.
datatypeThe datatype of the content.
uriURI of the content, if any.
xThe artifact object.
blobThe blob data. Can be a string or bytes. if string, it should be in the format &:blob:{file_id} if bytes, it should be the actual data.

Class for representing data to be saved on disk or in the artifact-store.

Blob​

Blob(self,
db: dataclasses.InitVar[typing.Optional[ForwardRef('Datalayer')]] = None,
uuid: None = <factory>,
*,
identifier: str,
bytes: bytes) -> None
ParameterDescription
identifierThe identifier of the blob.
dbDatalayer instance.
uuidUUID of the leaf.
bytesThe bytes of the blob.

A wrapper to signify a blob for special treatment.

See Document.encode and related functions.

DecodeTorchStateDict​

DecodeTorchStateDict(self,
cls)
ParameterDescription
clsTorch state cls

Torch state dictionary decoder.

Encodable​

Encodable(self,
db: dataclasses.InitVar[typing.Optional[ForwardRef('Datalayer')]] = None,
uuid: None = <factory>,
*,
identifier: str = '',
datatype: superduper.components.datatype.DataType,
uri: Optional[str] = None,
x: Any = <EMPTY>,
blob: dataclasses.InitVar[typing.Optional[bytearray]] = None) -> None
ParameterDescription
identifierIdentifier of the leaf.
dbDatalayer instance.
uuidUUID of the leaf.
datatypeThe datatype of the content.
uriURI of the content, if any.
xThe encodable object.
blobThe blob data.

Class for encoding non-Python datatypes to the database.

File​

File(self,
db: dataclasses.InitVar[typing.Optional[ForwardRef('Datalayer')]] = None,
uuid: None = <factory>,
*,
identifier: str = '',
datatype: superduper.components.datatype.DataType,
uri: Optional[str] = None,
x: Any = <EMPTY>) -> None
ParameterDescription
identifierIdentifier of the leaf.
dbDatalayer instance.
uuidUUID of the leaf.
datatypeThe datatype of the content.
uriURI of the content, if any.
xpath to the file

Data to be saved on disk and passed as a file reference.

FileItem​

FileItem(self,
db: dataclasses.InitVar[typing.Optional[ForwardRef('Datalayer')]] = None,
uuid: None = <factory>,
*,
identifier: str,
path: str) -> None
ParameterDescription
identifierThe identifier of the file.
dbDatalayer instance.
uuidUUID of the leaf.
pathThe path of the file.

File item class.

LazyArtifact​

LazyArtifact(self,
db: dataclasses.InitVar[typing.Optional[ForwardRef('Datalayer')]] = None,
uuid: None = <factory>,
*,
identifier: str = '',
datatype: superduper.components.datatype.DataType,
uri: Optional[str] = None,
x: Any = <EMPTY>,
blob: dataclasses.InitVar[typing.Union[str,
bytes,
NoneType]] = None) -> None
ParameterDescription
identifierIdentifier of the leaf.
dbDatalayer instance.
uuidUUID of the leaf.
datatypeThe datatype of the content.
uriURI of the content, if any.
xThe artifact object.
blobThe blob data. Can be a string or bytes. if string, it should be in the format &:blob:{file_id} if bytes, it should be the actual data.

Data to be saved and loaded only when needed.

LazyFile​

LazyFile(self,
db: dataclasses.InitVar[typing.Optional[ForwardRef('Datalayer')]] = None,
uuid: None = <factory>,
*,
identifier: str = '',
datatype: superduper.components.datatype.DataType,
uri: Optional[str] = None,
x: Any = <EMPTY>) -> None
ParameterDescription
identifierIdentifier of the leaf.
dbDatalayer instance.
uuidUUID of the leaf.
datatypeThe datatype of the content.
uriURI of the content, if any.
xpath to the file

Class is used to load a file only when needed.

Native​

Native(self,
db: dataclasses.InitVar[typing.Optional[ForwardRef('Datalayer')]] = None,
uuid: None = <factory>,
*,
identifier: str = '',
datatype: superduper.components.datatype.DataType,
uri: Optional[str] = None,
x: Optional[Any] = None) -> None
ParameterDescription
identifierIdentifier of the leaf.
dbDatalayer instance.
uuidUUID of the leaf.
datatypeThe datatype of the content.
uriURI of the content, if any.
xThe encodable object.

Class for representing native data supported by the underlying database.