com.google.cloud.hadoop.gcsio

Class GoogleCloudStorageFileSystem

java.lang.Object

com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystem

public class GoogleCloudStorageFileSystem
extends Object

Provides a POSIX like file system layered on top of Google Cloud Storage (GCS).

All file system aspects (eg, path) are encapsulated in this class, they are not exposed to the underlying layer. That is, all interactions with the underlying layer are strictly in terms of buckets and objects.

See Also:

Hadoop FileSystem specification.

Field Summary

Fields

Modifier and Type	Field and Description
static ListFileOptions	DELETE_RENAME_LIST_OPTIONS
static URI	GCS_ROOT
static String	SCHEME

Constructor Summary

Constructors

Constructor and Description
GoogleCloudStorageFileSystem(CheckedFunction<GoogleCloudStorageOptions,GoogleCloudStorage,IOException> gcsFn, GoogleCloudStorageFileSystemOptions options) Constructs a GoogleCloudStorageFilesystem based on an already-configured underlying GoogleCloudStorage gcs.
GoogleCloudStorageFileSystem(com.google.api.client.auth.oauth2.Credential credential, Function<List<AccessBoundary>,String> downscopedAccessTokenFn, GoogleCloudStorageFileSystemOptions options) Constructs an instance of GoogleCloudStorageFileSystem.
GoogleCloudStorageFileSystem(com.google.api.client.auth.oauth2.Credential credential, GoogleCloudStorageFileSystemOptions options) Constructs an instance of GoogleCloudStorageFileSystem.

Method Summary

Modifier and Type	Method and Description
void	close() Releases resources used by this instance.
void	compose(List<URI> sources, URI destination, String contentType) Composes inputs into a single GCS object.
WritableByteChannel	create(URI path) Creates and opens an object for writing.
WritableByteChannel	create(URI path, CreateFileOptions createOptions) Creates and opens an object for writing.
void	delete(URI path, boolean recursive) Deletes one or more items indicated by the given path.
boolean	exists(URI path) Indicates whether the given item exists.
FileInfo	getFileInfo(URI path) Gets information about the given path item.
List<FileInfo>	getFileInfos(List<URI> paths) Gets information about each path in the given list; more efficient than calling getFileInfo() on each path individually in a loop.
GoogleCloudStorage	getGcs() Retrieve our internal gcs.
GoogleCloudStorageFileSystemOptions	getOptions() Retrieve the options that were used to create this GoogleCloudStorageFileSystem.
List<FileInfo>	listFileInfo(URI path) If the given path points to a directory then the information about its children is returned, otherwise information about the given file is returned.
List<FileInfo>	listFileInfo(URI path, ListFileOptions listOptions) If the given path points to a directory then the information about its children is returned, otherwise information about the given file is returned.
List<FileInfo>	listFileInfoForPrefix(URI prefix) Equivalent to a recursive listing of prefix, except that prefix doesn't have to represent an actual object but can just be a partial prefix string.
List<FileInfo>	listFileInfoForPrefix(URI prefix, ListFileOptions listOptions) Equivalent to a recursive listing of prefix, except that prefix doesn't have to represent an actual object but can just be a partial prefix string.
GoogleCloudStorage.ListPage<FileInfo>	listFileInfoForPrefixPage(URI prefix, ListFileOptions listOptions, String pageToken) Equivalent to listFileInfoForPrefix(java.net.URI) but returns FileInfos listed by single request (1 page).
GoogleCloudStorage.ListPage<FileInfo>	listFileInfoForPrefixPage(URI prefix, String pageToken) Equivalent to listFileInfoForPrefix(java.net.URI) but returns FileInfos listed by single request (1 page).
void	mkdir(URI path) Creates a directory at the specified path.
void	mkdirs(URI path) Creates a directory at the specified path.
static CreateObjectOptions	objectOptionsFromFileOptions(CreateFileOptions options) Convert CreateFileOptions to CreateObjectOptions.
SeekableByteChannel	open(URI path) Opens an object for reading.
SeekableByteChannel	open(URI path, GoogleCloudStorageReadOptions readOptions) Opens an object for reading.
void	rename(URI src, URI dst) Renames the given item's path.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SCHEME

public static final String SCHEME

See Also:

Constant Field Values

GCS_ROOT

public static final URI GCS_ROOT

DELETE_RENAME_LIST_OPTIONS

public static final ListFileOptions DELETE_RENAME_LIST_OPTIONS

Constructor Detail

GoogleCloudStorageFileSystem

public GoogleCloudStorageFileSystem(com.google.api.client.auth.oauth2.Credential credential,
                                    GoogleCloudStorageFileSystemOptions options)
                             throws IOException

Constructs an instance of GoogleCloudStorageFileSystem.

Parameters:

credential - OAuth2 credential that allows access to GCS.

options - Options for how this filesystem should operate and configure its underlying storage.

Throws:

IOException

GoogleCloudStorageFileSystem

public GoogleCloudStorageFileSystem(com.google.api.client.auth.oauth2.Credential credential,
                                    Function<List<AccessBoundary>,String> downscopedAccessTokenFn,
                                    GoogleCloudStorageFileSystemOptions options)
                             throws IOException

Constructs an instance of GoogleCloudStorageFileSystem.

Parameters:

credential - OAuth2 credential that allows access to GCS.

downscopedAccessTokenFn - Function that generates downscoped access token.

options - Options for how this filesystem should operate and configure its underlying storage.

Throws:

IOException

GoogleCloudStorageFileSystem

public GoogleCloudStorageFileSystem(CheckedFunction<GoogleCloudStorageOptions,GoogleCloudStorage,IOException> gcsFn,
                                    GoogleCloudStorageFileSystemOptions options)
                             throws IOException

Constructs a GoogleCloudStorageFilesystem based on an already-configured underlying GoogleCloudStorage gcs. Any options pertaining to GCS creation will be ignored.

Throws:

IOException

Method Detail

getOptions

public GoogleCloudStorageFileSystemOptions getOptions()

Retrieve the options that were used to create this GoogleCloudStorageFileSystem.

objectOptionsFromFileOptions

public static CreateObjectOptions objectOptionsFromFileOptions(CreateFileOptions options)

Convert CreateFileOptions to CreateObjectOptions.

create

public WritableByteChannel create(URI path)
                           throws IOException

Creates and opens an object for writing. If the object already exists, it is deleted.

Parameters:

path - Object full path of the form gs://bucket/object-path.

Returns:

A channel for writing to the given object.

Throws:

IOException

create

public WritableByteChannel create(URI path,
                                  CreateFileOptions createOptions)
                           throws IOException

Creates and opens an object for writing.

Parameters:

path - Object full path of the form gs://bucket/object-path.

Returns:

A channel for writing to the given object.

Throws:

IOException

open

public SeekableByteChannel open(URI path)
                         throws IOException

Opens an object for reading.

Parameters:

path - Object full path of the form gs://bucket/object-path.

Returns:

A channel for reading from the given object.

Throws:

FileNotFoundException - if the given path does not exist.

IOException - if object exists but cannot be opened.

open

public SeekableByteChannel open(URI path,
                                GoogleCloudStorageReadOptions readOptions)
                         throws IOException

Opens an object for reading.

Parameters:

path - Object full path of the form gs://bucket/object-path.

readOptions - Fine-grained options for behaviors of retries, buffering, etc.

Returns:

A channel for reading from the given object.

Throws:

FileNotFoundException - if the given path does not exist.

IOException - if object exists but cannot be opened.

delete

public void delete(URI path,
                   boolean recursive)
            throws IOException

Deletes one or more items indicated by the given path.

If path points to a directory:

• if recursive is true, all items under that path are recursively deleted followed by deletion of the directory.
• else,
  • the directory is deleted if it is empty,
  • else, an IOException is thrown.

The recursive parameter is ignored for a file.

Parameters:

path - Path of the item to delete.

recursive - If true, all sub-items are also deleted.

Throws:

FileNotFoundException - if the given path does not exist.

IOException

exists

public boolean exists(URI path)
               throws IOException

Indicates whether the given item exists.

Parameters:

path - Path of the item to check.

Returns:

true if the given item exists, false otherwise.

Throws:

IOException

mkdirs

public void mkdirs(URI path)
            throws IOException

Creates a directory at the specified path. Also creates any parent directories as necessary. Similar to 'mkdir -p' command.

Parameters:

path - Path of the directory to create.

Throws:

IOException

rename

public void rename(URI src,
                   URI dst)
            throws IOException

Renames the given item's path.

The operation is disallowed if any of the following is true:

• src equals GCS_ROOT
• src is a file and dst equals GCS_ROOT
• src does not exist
• dst is a file that already exists
• parent of the destination does not exist

Otherwise, the expected behavior is as follows:

• if src is a directory:
  • if dst is an existing file then disallowed
  • if dst is a directory then rename the directory

• if src is a file:
  • if dst is a file then rename the file.
  • if dst is a directory then similar to the previous case after appending src file-name to dst

Note: This function is very expensive to call for directories that have many sub-items.

Parameters:

src - Path of the item to rename.

dst - New path of the item.

Throws:

FileNotFoundException - if src does not exist.

IOException

compose

public void compose(List<URI> sources,
                    URI destination,
                    String contentType)
             throws IOException

Composes inputs into a single GCS object. This performs a GCS Compose. Objects will be composed according to the order they appear in the input. The destination object, if already present, will be overwritten. Sources and destination are assumed to be in the same bucket.

Parameters:

sources - the list of URIs to be composed

destination - the resulting URI with composed sources

contentType - content-type of the composed object

Throws:

IOException - if the Compose operation was unsuccessful

listFileInfoForPrefix

public List<FileInfo> listFileInfoForPrefix(URI prefix)
                                     throws IOException

Equivalent to a recursive listing of prefix, except that prefix doesn't have to represent an actual object but can just be a partial prefix string. The 'authority' component of the prefix must be the complete authority, however; we can only list prefixes of objects, not buckets.

Parameters:

prefix - the prefix to use to list all matching objects.

Throws:

IOException

listFileInfoForPrefix

public List<FileInfo> listFileInfoForPrefix(URI prefix,
                                            ListFileOptions listOptions)
                                     throws IOException

Equivalent to a recursive listing of prefix, except that prefix doesn't have to represent an actual object but can just be a partial prefix string. The 'authority' component of the prefix must be the complete authority, however; we can only list prefixes of objects, not buckets.

Parameters:

prefix - the prefix to use to list all matching objects.

Throws:

IOException

listFileInfoForPrefixPage

public GoogleCloudStorage.ListPage<FileInfo> listFileInfoForPrefixPage(URI prefix,
                                                                       String pageToken)
                                                                throws IOException

Equivalent to listFileInfoForPrefix(java.net.URI) but returns FileInfos listed by single request (1 page).

Parameters:

prefix - the prefix to use to list all matching objects.

pageToken - the page token to list

Throws:

IOException

listFileInfoForPrefixPage

public GoogleCloudStorage.ListPage<FileInfo> listFileInfoForPrefixPage(URI prefix,
                                                                       ListFileOptions listOptions,
                                                                       String pageToken)
                                                                throws IOException

Equivalent to listFileInfoForPrefix(java.net.URI) but returns FileInfos listed by single request (1 page).

Parameters:

prefix - the prefix to use to list all matching objects.

pageToken - the page token to list

Throws:

IOException

listFileInfo

public List<FileInfo> listFileInfo(URI path)
                            throws IOException

If the given path points to a directory then the information about its children is returned, otherwise information about the given file is returned.

Parameters:

path - Given path.

Returns:

Information about a file or children of a directory.

Throws:

FileNotFoundException - if the given path does not exist.

IOException

listFileInfo

public List<FileInfo> listFileInfo(URI path,
                                   ListFileOptions listOptions)
                            throws IOException

If the given path points to a directory then the information about its children is returned, otherwise information about the given file is returned.

Parameters:

path - Given path.

Returns:

Information about a file or children of a directory.

Throws:

FileNotFoundException - if the given path does not exist.

IOException

getFileInfo

public FileInfo getFileInfo(URI path)
                     throws IOException

Gets information about the given path item.

Parameters:

path - The path we want information about.

Returns:

Information about the given path item.

Throws:

IOException

getFileInfos

public List<FileInfo> getFileInfos(List<URI> paths)
                            throws IOException

Gets information about each path in the given list; more efficient than calling getFileInfo() on each path individually in a loop.

Parameters:

paths - List of paths.

Returns:

Information about each path in the given list.

Throws:

IOException

close

public void close()

Releases resources used by this instance.

mkdir

public void mkdir(URI path)
           throws IOException

Creates a directory at the specified path.

There are two conventions for using objects as directories in GCS. 1. An object of zero size with name ending in / 2. An object of zero size with name ending in _$folder$

#1 is the recommended convention by the GCS team. We use it when creating a directory.

However, some old tools still use #2. We will decide based on customer use cases whether to support #2 as well. For now, we only support #1.

Note that a bucket is always considered a directory. Doesn't create parent directories; normal use cases should only call mkdirs().

Throws:

IOException

getGcs

public GoogleCloudStorage getGcs()

Retrieve our internal gcs.