It Appears That You Are Directly Uploading Binary Data in an Unrecognized Format .bam

File Format Guide

• Introduction
• BAM files
• CRAM files
• SFF files
• HDF5 files
  • PacBio
  • MinION Oxford Nanopore
  • HDF5 tools
• FASTQ files
  • Paired-terminate FASTQ
  • Platform specific FASTQ files
    • 454 fastq
    • Ion Torrent fastq
    • Contempo Illumina fastq
    • Older Illumina fastq
    • QIIME de-multiplexed sequences in fastq
    • PacBio CCS (Circular Consensus Sequence) or RoI (Read of Insert) read
    • PacBio CCS subread
    • Helicos fastq with a fixed ASCII-based Phred value for quality
    • FASTA files
• FASTA with QUAL file pairs
• CSFASTA with QUAL Files
• Legacy Formats
  • SRF files
  • Native Illumina
  • QSEQ
• Car Specific Information
  • Illumina
  • SOLiD
  • Roche 454 (formerly Life Sciences)
  • IonTorrent
  • PacBio
  • MinION Oxford Nanopore
  • Helicos
  • Capillary (Sanger)
  • CompleteGenomics
• Contact SRA

Introduction

This page reviews the submission file formats currently supported by the Sequence Read Athenaeum (SRA) at NCBI, EBI, and DDBJ, and gives guidance to submitters about current and future file formats and policies regarding SRA submissions.

Some things to keep in listen:

• The SRA is a raw information annal, and requires per-base of operations quality scores for all submitted data. Therefore, FASTA and other sequence-only formats are not sufficient for submission! FASTA can, however, be submitted as a reference sequence(s) for BAM files or every bit function of a FASTA/QUAL pair (run into beneath).
• SRA accepts binary files such equally BAM, SFF, and HDF5 formats and text formats such as FASTQ.

BAM files

Binary Alignment/Map files (BAM) represent one of the preferred SRA submission formats. BAM is a compressed version of the Sequence Alignment/Map (SAM) format (see SAMv1 (.pdf)). BAM files tin be decompressed to a homo-readable text format (SAM) using SAM/BAM-specific utilities (e.g. samtools ) and can comprise unaligned sequences besides. SRA recommends adjustment to an unmodified known reference, if possible, to enable subsequent users to view the alignments in the Sequence Viewer or to compare the alignments with other alignments on the aforementioned reference.

SAM is a tab-delimited format including both the raw read data and information about the alignment of that read to a known reference sequence(due south). At that place are ii main sections in a SAM file, the header and the alignment (sequence read) sections, each of which are described beneath. Note that this documentation will focus on a description of the SAM format with respect to submission of BAM files to the SRA (i.e. SRA doe not take SAM files for submission). A more comprehensive discussion of the format specifications tin exist found at the samtools website.

SAM Header Example:

@Hd    VN:ane.4    And so:coordinate
@SQ    SN:CHROMOSOME_I    LN:15072423
UR:ftp://ftp.ncbi.nlm.nih.gov/genbank/genomes/Eukaryotes/invertebrates/Caenorhabditis_elegans/
WBcel215/Primary_Assembly/assembled_chromosomes/FASTA/chrI.fa.gz    AS:ce10
SP:Caenorhabditis elegans

@SQ    SN:CHROMOSOME_II    LN:15279345
UR:ftp://ftp.ncbi.nlm.nih.gov/genbank/genomes/Eukaryotes/invertebrates/Caenorhabditis_elegans/
WBcel215/Primary_Assembly/assembled_chromosomes/FASTA/chrII.fa.gz     AS:ce10
SP:Caenorhabditis elegans

@RG    ID:1    PL:ILLUMINA    LB:C_ele_05    DS:WGS of C elegans    PG:BamIndexDecoder
@PG    ID:bwa    PN:bwa    VN:0.v.10-tpx

Ideally, the SN value should be a versioned accession (due east.grand., NC_003279.7 , rather than CHROMOSOME_I ). This volition allow the SRA to unambiguously identify the reference sequence(south) and process the BAM file with minimal intervention. Otherwise, submitters are strongly encouraged to include the "URL/URI" that can be used to obtain the reference sequence(due south) and Equally tags to clearly define which assembly has been used (equally higher up).

If the data are instead aligned to a local or submitter-defined set of references (including any modifications to accessioned assemblies), then the submitter must include a reference fasta forth with each submitted bam file. Note: the FASTA header line(s) MUST match the SN names provided in the BAM file exactly.

Deviation from these recommended practices will require manual intervention past SRA staff in order to process a BAM file and tin filibuster completion of a submission and acquisition of accession numbers.

SAM Alignment Case:

3658435    145    CHROMOSOME_I    one    0    100M    CHROMOSOME_II    2716898    0
GCCTAAGCCTAAGCCTAAGCCTAAGCCTAAGCCTAAGCCTAAGCCTAAGCCTAAGCCTAAGCCTAAGCCTAAGCCTAAGCCTAAGCCTAAGCCT
AAGCCT
@CCC?:CCCCC@CCCEC>AFDFDBEGHEAHCIGIHHGIGEGJGGIIIHFHIHGF@HGGIGJJJJJIJJJJJJJJJJJJJJJJJJJJJHHHHHFF
FFFCCC    RG:Z:i    NH:i:1    NM:i:0

5482659    65    CHROMOSOME_I    1    0    100M    CHROMOSOME_II    11954696    0
GCCTAAGCCTAAGCCTAAGCCTAAGCCTAAGCCTAAGCCTAAGCCTAAGCCTAAGCCTAAGCCTAAGCCTAAGCCTAAGCCTAAGCCTAAGCCT
AAGCCT
CCCFFFFFHHGHGJJGIJHIJIJJJJJIJJJJJIJJGIJJJJJIIJIIJFJJJJJFIJJJJIIIIGIIJHHHHDEEFFFEEEEEDDDDCDCCCA
AA?CC:    RG:Z:one    NH:i:1    NM:i:0

The header and alignment section are internally consistent: each aligned read has an RNAME (reference sequence name, 3^rd field) that matches an SN tag value from the header (e.g., CHROMOSOME_I ), and, if provided, the alignment read group optional field ( RG:Z: ) is consistent with the read group ID in the header ( 1 ). It is also important to ensure that the FLAG fields (2^nd field in each line) are correctly set for the data. The SRA pipeline will attempt to resolve wrong FLAG values, merely sufficiently incorrect values can lead to processing errors. The SRA does not archive optional and not-standard tags/field values independent in the alignment section. Nonetheless, the entire header department of the bam file is retained. Additionally, although the SAM format allows for an equal sign (=) in the sequence field to correspond a match to the reference sequence or only an asterisk (*) in both the sequence and quality fields, the SRA processing software does not recognize either of these formats.

Please note that unexpected notations used to indicated paired reads tin pb to failure to recognize the pairs and an improper SRA archive (i.e. paired reads are treated similar fragments). For example, using :0 and :1 at the finish of the read names is singular and is currently non recognized equally an indication of read 1 and 2 in a pair. It would be amend to exclude these notations and provide the 2 reads with the same names. Expected notations for item platforms volition work. For example, Illumina reads with /1 or /2 appended is an expected note. Further, neglecting to set the proper $.25 for paired reads in the SAM/BAM flags (e.thou. multi-segment template 1-bit, first segment 64-bit, and last segment 128-bit) or splitting paired reads into separate bam files tin event in an improper SRA archive or failure to generate the SRA archive.

When submitting BAM files of aligned reads to the SRA you must also specify an assembly - the reference genome that your reads were aligned against. You tin identify your reference assembly past its proper name or accession from the NCBI Assembly database. UCSC and Ensembl associates names may likewise be used. If the assembly is not bachelor from a public repository you will need to submit your own (local) assembly in FASTA format (reference_fasta) forth with your BAM file.

CRAM files

Another adequate SRA submission format is the CRAM format (see CRAMv3(.pdf)). Files received in this format are converted to the BAM format for processing. The references provided in this format are treated in the aforementioned manner every bit BAM references with the added possibility of a cheque against the European Nucleotide Archive (ENA) CRAM reference registry.

SFF files

In the absence of a BAM file, Standard Flowgram Files or SFF is the preferred input format for 454 Life Sciences (at present office of Roche) data; IonTorrent data can as well be submitted as SFF. All-encompassing technical details almost the format can be obtained here .

Submitters of SFF data should ensure that the information are demultiplexed (if barcoded) – this is particularly common in pyrotag / 16S rRNA amplicon sequencing.

HDF5 files

HDF5 is a data model, library, and file format for storing and managing data. The SRA accepts bas.h5 and bax.h5 file submissions for PacBio-based submission and .fast5 files for submissions related to MinION Oxford Nanopore.

PacBio

Submission of data from the RS Ii instrument requires one (1) bas.h5 file and three (3) bax.h5 files. Practise not link more than ane PacBio RS II to an SRA run and please do not change the bax.h5 files names from those indicated in the bas.h5 file.

Depending on the platform used for your PacBio sequencing projection, the post-obit information files with corresponding extensions are produced and required for SRA submission.

PacBio RS Platform	Data Files Delivered
PacBio RS	xxxx.metadata.xml (optional simply desirable) xxxx.bas.h5
PacBio RS Two	xxxx.metadata.xml (optional merely desirable) xxxx.bas.h5 (optional merely desirable) xxxx.i.bax.h5 xxxx.2.bax.h5 xxxx.3.bax.h5

Delight be sure to list the files for each SMRT Prison cell in a divide Run or on a divide row of your sra_metadata sail.

PacBio documentation on bax.h5 / bas.h5 format: bas.h5ReferenceGuide.pdf.

MinION Oxford Nanopore

In this instance, in that location are 1-3 sequences per fast5 HDF file (1 spot of information) and the entire prepare of fast5 files should be submitted in a tar.gz file. You must submit the fast5 files generated later on base calling.

Learn more about this platform at Oxford Nanopore Technologies website.

HDF5 tools

HDF5 tools: http://world wide web.hdfgroup.org/products/hdf5_tools

FASTQ files

Fastq consists of a defline that contains a read identifier and perhaps other information, nucleotide base calls, a second defline, and per-base quality scores, all in text form. There are many variations.

The following terms and formats are divers in general:

• Identifier and other information: text string terminated by white space.
• Bases: fastq sequence should contain standard base calls (ACTGactg) or unknown bases (Nn) and tin can vary in length.

• Qualities options:

  Decimal-encoding, space-delimited	[0-9]+ | <quality>\due south[0-ix]+
  Phred-33 ASCII	[\!\"\#\$\%\&\'\(\)\*\+,\-\.\/0-9:;<=>\?\@A-I]+
  Phred-64 ASCII	[\@A-Z\[\\\]\^_`a-h]+

  Quality string length should be equal to sequence length.

  In a limited set of cases, log odds or non-ASCII numerical quality values will succeed during an SRA submission.

Files from diverse platforms employing this format are adequate:

@<identifier and expected information>
<sequence>
+<identifier and other data OR empty string>
<quality>

Where each instance of Identifier, Bases, and Qualities are newline-separated. Extra information added beyond the <identifier and expected information> examples is likely to be discarded/ignored.

As indicated higher up, the Qualities string can be space-separated numeric Phred scores or an ASCII string of the Phred scores with the ASCII character value = Phred score plus an kickoff constant used to place the ASCII characters in the printable graphic symbol range. There are 2 predominant offsets: 33 (0 = !) and 64 (0=@).

Paired-end FASTQ

Although generally the case, there are some instances where paired reads are not a forrard read paired with a reverse read.

Paired-stop information submitted in FASTQ format should be submitted in 1 of two formats:

1. As split up files for forrad and reverse reads, in which the reads are in the same gild.
2. As interleaved, or "viii-line", FASTQ, in which forward and reverse reads alternating in the file and are in order (i.e., read "1F", followed by read "1R", then read "2F", so "2R").

SRA supports the following forward/reverse read indicators: '/1' and '/two' at the end of the read name or newer Illumina style '1:Y:18:ATCACG' and '2:Y:18:ATCACG'.

Concatenated FASTQ (in which all forward reads are followed by all reverse reads) is non supported.

Platform specific FASTQ files

454 fastq

@<454_universal_accession>

Under Roche 454, SRA accepts both 'pre-split up' or 'post-split' 454 fastq sequences. Paired 'post-split' 454 reads must be provided in split up files or in the interleaved format. 'Split' ways the 454 linker has been located/removed and used to split the sequence into biological read pairs (and all other technical reads take been removed).

Ion Torrent fastq

@<Run_ID>:<Chip_Row_Coordinate>:<Chip_Column_Coordinate>

In the same manner equally Roche 454, SRA simply accepts 'pre-split' Ion Torrent sequences or 'post-carve up' Ion Torrent single read fragments in a fastq form. Paired 'mail service-separate' Ion Torrent reads will require submission in a BAM file. 'Split' ways the Ion Torrent linker has been located/removed and used to split the sequence into biological read pairs (and all other technical reads have been removed).

Recent Illumina fastq

@<instrument>:<run number>:<flowcell ID>:<lane>:<tile>:<xpos>:<y-pos> <read>:<is filtered>:<control number>:<index>

<alphabetize> values for Illumina fastq can exist barcodes.

Older Illumina fastq

@<machine_id>:<lane>:<tile>:<x_coord>:<y_coord>#<index>/<read>

<index> values for Illumina fastq can be barcodes.

QIIME de-multiplexed sequences in fastq

@<SampleID-based_identifier> <Original_information> orig_bc=<original_barcode> new_bc=<corrected_barcode> bc_diffs=<0|ane>

PacBio CCS (Circular Consensus Sequence) or RoI (Read of Insert) read

@<MovieName>/<ZMW_number>

PacBio CCS subread

@<MovieName> /<ZMW_number>/<subread-kickoff>_<subread-end>

Helicos fastq with a fixed ASCII-based Phred value for quality

@VHE-242383071011-fifteen-1-0-2

Characteristic use of a quality '/', which gives a Phred value of 14.

The native format for helicos is fasta and so converting to fastq requires creating a default quality score. The default value selected by the SRA squad is '14'.

FASTA files

Fasta files adhering to the definition lines described in the fastq section are acceptable, likewise, although fastq is preferred (a file blazon of fastq should still be specified). The SRA assigns a default quality value of thirty in this case and expects this format:

>(identifier and other information)
<sequence>

FASTA with QUAL file pairs

Fasta files may be submitted with respective qual files, also. These are recognized in the SRA data processing pipeline every bit equivalent to fastq and should be specified as fastq when submitting the data files.

Files from some platforms (by and large older Illumina and Roche 454) employing this format are acceptable and the entries in the pair of files should look similar:

File 1

>READNAME
BASES

File 2

>READNAME
QUALITIES

Where READNAME must be identical between files for a given read, and QUALITIES are generally in whitespace-separated decimal values.

Annotation the following guidelines for FASTA/QUAL pairs of files:

In a given pair of files, there must be the same number of reads in both. For a given read, at that place must be the same number of BASES and QUALITIES, i.e., if the BASES are trimmed to remove barcodes, then the aforementioned scores must be removed from the QUALITIES, etc.

CSFASTA with QUAL Files

The files have an optional header that is identified past lines that begin with the hash/pound/number sign (#). The HEADER can exist defined every bit:

# <date> <path> [--flag]* --tag <tag> --minlength=<length> --prefix=<prefix> <path>
# Cwd: <path>
# Title: <flowcell>

The permissible CSFASTA format is equally follows:

#HEADER (multiple lines)
>TAGNAME
BASES

The permissible QUAL format is every bit follows:

#HEADER (multiple lines)
>TAGNAME
QUALITIES

Every bit with FASTA/QUAL pairs, there are several rules for pairs of CSFASTA/QUAL files. TAGNAME must be identical between files for a given read, and QUALITIES are generally in whitespace-separated decimal values.

Note the following guidelines for CSFASTA/QUAL pairs of files:

In a given pair of files, at that place must be the same number of reads in both. For a given read, there must be the aforementioned number of colour infinite digits and QUALITIES, i.e., the BASES line is typically i character longer than the number of QUALITIES (due to the color space indexing base of operations that begins each BASES string). HEADER must exist identical between paired files.

Too see SOLiD™ Information Format and File Definitions Guide (.pdf)

Legacy Formats

These formats are nonetheless accepted past SRA, but are considered out-of-date and not recommended for submission. If you are able to update your files to a more mutual format please do and then before submitting to SRA.

SRF files

SRF is a generic format for Deoxyribonucleic acid sequence information. This format has sufficient flexibility to store data from current and future Deoxyribonucleic acid sequencing technologies. This is a single input file format for all downstream applications and a read lookup index enabling downstream formats to reference reads without duplication of all of the read specific information.

Sequence Read Format (SRF) homepage: http://srf.sourceforge.internet/ .

Native Illumina

Submitters may submit native information from the primary analysis output of the Illumina GA.

The filetype is Illumina_native and constituent files for a run should exist tarred together into a unmarried tar file.

Illumina GA readname can exist defined as follows:

<flowcell> = [a-zA-Z0-9_-]{2}+
<lane> = 1..8
<title> = i..1024
<X> = 1..4096
<Y> = i..4096
<sep> ::= [_\t]
READNAME ::= [<flowcell><sep> | s_]<lane><sep><tile><sep><10><sep><y>

Inside a related ready of files, reads are grouped by tile. Reads should be fixed length, and the number of quality scores and bases is the same in each.

Immune characters:

BASES: AaCcTtGgNn

QUALITIES: \!\"\#\$\%\&\'\(\)\*\+,\-\.\/0-9:;<=>\?\@A-I]+ or \@A-Z\[\\\]\^_`a-h]+

QSEQ

The basecalling program Bustard emits a _qseq.txt file for each lane (two files for mate pairs). Paired-end information are presented in the orientation in which they were sequenced (five'-3'& 3'-v').

Each read is contained on a single line with tab separators in the post-obit format:

• Machine proper name: Unique identifier of the sequencer.
• Run number: Unique number to identify the run on the sequencer.
• Lane number: Positive Integer (currently 1-viii).
• Tile number: Positive Integer.
• X coordinate of the spot: Integer (tin exist negative).
• Y coordinate of the spot: Integer (can exist negative).
• Index: Positive Integer (no indexing should take a value of 1).
• Read Number: 1 for single reads; 1 or two for paired-ends.
• Sequence (BASES)
• Quality: the calibrated quality string (QUALITIES).
• Filter: Did the read pass filtering? 0 - No, i - Yes.

Machine Specific Information

File types accustomed past platform in approximate order of preference (formats that are least desirable marked with '*', those with uncertain effect marked with '?'):

Illumina

bam, fastq, qseq, fasta+qual^*?, native^*, srf^*?

SOLiD

bam, csfasta + QV.qual, srf^*?

Roche 454 (formerly Life Sciences)

bam, sff, fastq, fasta+qual^*?

IonTorrent

bam, sff, fastq, fasta+qual^*?

PacBio

bam, hdf5, fastq

MinION Oxford Nanopore

hdf5, fastq

Helicos

bam, fastq

Capillary (Sanger)

bam, fastq^*?

CompleteGenomics

native, bam^*

Complete Genomics format – run into CG Data File Formats . This format requires providing tarred versions of the ASM, LIB, and MAP sub-directories for a successful submission to have place. Additionally, processing of reference sequences occurs in the aforementioned manner equally for BAM and CRAM files. For this format, please contact SRA prior to submission.

---

Contact SRA staff for assistance at sra@ncbi.nlm.nih.gov

phillipsmothough.blogspot.com

Source: https://www.ncbi.nlm.nih.gov/sra/docs/submitformats/

Share this post