
<resource schema="cars">
	<meta name="title">CARS survey data</meta>
	<meta name="creationDate">2009-01-20T10:40:00</meta>
	<meta name="schema-rank">1000</meta>
	<meta name="description">Images and data from from the CFHTLS archive
		research survey, a multi-band dataset spanning 37 square degrees of
		sky in high galactic latitudes.</meta>

	<meta name="subject">catalogs</meta>
	<meta name="subject">surface-photometry</meta>
	<meta name="subject">galaxy-distances</meta>
	<meta name="subject">observational-cosmology</meta>

	<meta name="source">2009A&amp;A...493.1197E</meta>
	<meta name="creator">Erben, T.; Hildebrandt, H.; Lerchster, M.; Hudelot,
		P.; Benjamin, J.; van Waerbeke, L.; Schrabback, T.; Brimioulle, F.; Cordes,
		O.; Dietrich, J. P.; Holhjem, K.; Schirmer, M.; Schneider, P.</meta>
	<meta name="copyright" format="rst"><![CDATA[If you use the CARS products for your own research,
		please give proper acknowledgment to
		`CADC <http://www4.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/>`_, the
		`CFHTLS project <http://www.cfht.hawaii.edu/Science/CFHLS/cfhtlspublitext.html>`_ and the
		publication 2009A&A...493.1197E]]></meta>
	<meta name="_longdoc" format="rst"><![CDATA[
These services deliver products from the CFHTLS-Archive-Research Survey
(CARS). The survey currently offers derived products from 37
sq. degrees of the Wide part from the Canada-France-Hawaii Telescope
Legacy Survey (CFHTLS). The main documentation for the products, its
creation and its content is the publication 2009A&A...493.1197E.	
Here, we discuss some points which are not evident from this publication.

AVAILABLE DATA PRODUCTS
=======================

For each 1 sq. deg. CARS pointing we currently offer the following products
(we discuss them on the example of the CARS pointing W1m0p1); a typical file
is named W1m0p1_izrgu.V1.7A.final.cat. The name contains the following
information:

* W1m0p1 --	name of CARS field under consideration
* izrgu --  filters involved in the product (here i', r', z', g', u*);
  for catalogues etc. the first listed filter corresponds to
  the object detection filter (here 'i'; see also 2009A&A...493.1197E)
* V1.7A --		internal CARS version number
* final --		'final' product

The individual files we offer are:

W1m0p1_izrgu.V1.7A.final.cat
	object catalogue in FITS binary format; beware that they are very
	big and cumbersome to handle if you do not have efficient tools to
	work on big FITS binary tables!	(typical file size per field: 0.5GB!)

W1m0p1_izrgu.V1.7A.final.asc.gz
	An ASCII version of W1m0p1_izrgu.V1.7A.final.cat containing a subset
	of the most important catalogue quantities. This catalogue should be
	sufficient for most users.	The main content of the catalogue can
	also be queried in the \RSTservicelink{/cars/q/cat/form}{cat service}.
	(typical gzipped file size per field: 20MB)

W1m0p1_izrgu.V1.7A.final.reg.gz
	An ASCII ds9 region file containing saoimage/ds9 polygon entries
	(in pixel coordinates). They represent the CARS image masks described
	in Sect. 3.4 of 2009A&A...493.1197E. Polygons originating from different
	masking procedures are encoded with different colours (look at
	expressions such as '# color=yellow' at the end of each line).
	We offer this file so that interested users can construct own masks
	containing only parts of our global MASK key. (typical gzipped file size
	per field: 1MB)

	The colour coding in these files is as follows:

	- yellow:	masks from our object density analysis and stars identified
	  in the USNO-A2 catalogue
	- green:	 stars identified in the GSC1 and GSC2 cataogues
	- magenta: masks around extended haloes of bright stars
	- red:		 masks from our asteroid track analysis

W1m0p1_irg.V1.7A.truecol.jpg

	low resolution true colour i', r' and g' JPG image of the CARS field.
	These images allow a quick look at the data but they are probably not
	suitable for high-quality poster print-outs.	(typical file size per field:
	10MB)

W1m0p1_izrgu.V1.7A.masks.obj.jpg
	low resolution JPG image showing object positions and object masks
	overlayed on the i'-band image. It allows a quick-look at field-layout
	and object positions.	(typical file size per field: 3MB)

	The colour coding in this is:

	- green:	areas excluded from scientific analysis by our MASK key
	- yellow: stellar sources (SExtractor CLASS_STAR >=0.95)
	- red: galaxies with high-confidence photo-z estimates:
	  (SExtractor CLASS_STAR < 0.95; reliable object photometry
	  in five bands; BPZ ODDS quality parameter > 0.9);
	  see also Sect. 4 of 2009A&A...493.1197E
	- blue:	 all other galaxies (SExtractor CLASS_STAR < 0.95)

CATALOGUE QUANTITIES
====================

See 2009A&A...493.1197E, in particular Sect. 4, and the SExtractor Users manual
for a description of most catalogue entries.  Also see the table notes.

CURRENT COVERAGE
================

.. image:: \internallink{cars/q/cat/static/fields.png}
	 :alt: CARS coverage
	]]></meta>

	<meta name="coverage">
		<meta name="waveband">Optical</meta>
		<meta name="waveband">Infrared</meta>
	</meta>

	<table id="srccat" onDisk="True" adql="True" primary="carsid"
			mixin="//scs#q3cindex">
		<mixin>//scs#pgs-pos-index</mixin>
		<meta name="description">Extracted sources from the CARS survey,
			comprising positions and multiband photometry.</meta>
		<stc>Position ICRS "raj2000" "dej2000"</stc>
		<index columns="magAuto"/>
		<index columns="fwhm"/>
		<column name="carsid" description="CARS object ID" tablehead="Id"
			type="text" verbLevel="5" ucd="meta.id;meta.main"/>
		<column name="SeqNr" type="integer" required="True"
			description="Running object number (in field)"
			tablehead="No" ucd="meta.id"
			verbLevel="15"/>
		<column name="raj2000" description="Right ascension of barycenter (J2000)"
			tablehead="RA" type="double precision" ucd="pos.eq.ra;meta.main"
			verbLevel="1" unit="deg" displayHint="type=hms"/>
		<column name="dej2000" description="Declination of barycenter (J2000)"
			tablehead="Dec" type="double precision" ucd="pos.eq.dec;meta.main"
			verbLevel="1" unit="deg" displayHint="type=dms"/>
		<column name="xpos" description="Object position on exposure along x"
			tablehead="X" verbLevel="25" unit="pix" ucd="pos.cartesian.x;instr.det"/>
		<column name="ypos" description="Object position on exposure along y"
			tablehead="Y" verbLevel="25" unit="pix" ucd="pos.cartesian.y;instr.det"/>
		<column name="magAuto" unit="mag" ucd="phot.mag" verbLevel="25"
			description="Kron-like elliptical aperture magnitude"
			tablehead="Auto. mag." displayHint="sf=2" note="1"/>
		<column name="magErrAuto" unit="mag" ucd="stat.error;phot.mag"
			verbLevel="25" tablehead="Err. Auto. mag."
			description="Error in Kron-like elliptical aperture magnitude"
			displayHint="sf=2"/>
		<LOOP>
			<csvItems>
				bandS,band
				U,u*
				G,g'
				R,r'
				I,i'
				Z,z'
			</csvItems>
			<events>
				<column name="magiso\bandS" unit="mag" ucd="phot.mag;em.opt"
					description="Isophotal magnitude in the \band band"
					tablehead="Iso. mag. \band" verbLevel="9"
					displayHint="sf=2" note="1"/>
				<column name="magisoErr\bandS" unit="mag"
					ucd="stat.error;phot.mag;em.opt"
					description="RMS error of isophotal magnitude in the \band band"
					tablehead="Err. iso. mag. \band" verbLevel="19"
					displayHint="sf=2"/>
				<column name="magapr\bandS" unit="mag" ucd="phot.mag;em.opt"
					description="Fixed aperture magnitude in the \band band"
					tablehead="F.a. mag. \band" verbLevel="9"
					displayHint="sf=2" note="1"/>
				<column name="magaprErr\bandS" unit="mag"
					ucd="stat.error;phot.mag;em.opt"
					description="RMS error of fixed aperture magnitude in the \band band"
					tablehead="Err. F.a. mag. \band" verbLevel="19"
					displayHint="sf=2"/>
				<column name="maglim\bandS" unit="mag"
					description="Limiting magnitude at object's position in \band band"
					verbLevel="25" tablehead="Lim. mag. \band"
					displayHint="sf=2" note="4"/>
			</events>
		</LOOP>
		<column name="fwhm" unit="deg" tablehead="FWHM" verbLevel="15"
			description="Full width half max assuming a gaussian core"
			ucd="phys.angSize" displayHint="displayUnit=arcsec,sf=2"/>
		<column name="fluxRadius" unit="pix" description="Fraction-of-light radius"
			tablehead="Flux rad." verbLevel="21"/>
		<column name="majorAxis" unit="deg" ucd="phys.angSize.smajAxis"
			description="Profile RMS along major axis" verbLevel="15"
			tablehead="RMS major"/>
		<column name="minorAxis" unit="deg" ucd="phys.angSize.sminAxis"
			description="Profile RMS along minor axis" verbLevel="15"
			tablehead="RMS minor"/>
		<column name="theta" unit="deg" ucd="pos.posAng"
			description="Position angle (east of north) (J2000)"
			tablehead="Pos. Ang." verbLevel="15"/>
		<column name="sgclass" ucd="src.class.starGalaxy" tablehead="S/G"
			description="S/G classifier output (0..1)" verbLevel="15"
			displayHint="sf=2"/>
		<column name="extflags" type="integer" required="True"
			tablehead="Ext. Flags"
			description="Extraction Flags" verbLevel="29"/>
		<column name="mask" ucd="meta.code" required="True"
			tablehead="In mask"
			description="True if object is within an object mask"
			type="boolean" verbLevel="15"/>
		<column name="z_b" tablehead="Z_B" description=
			"Photometric redshift" verbLevel="29"/>
		<column name="t_b" tablehead="T_B"
			description="Mask value" verbLevel="29"/>
		<column name="odds" tablehead="z odds" description=
			"Confidence parameter for photometric redshift" verbLevel="29"/>
		<column name="NgoodFilters" type="integer" required="True"
			description="Number of filters with good photometry for BPZ"
			tablehead="#Good" verbLevel="29" ucd="meta.code.qual" note="2"/>
		<column name="NbadFilters" type="integer" required="True"
			description="Number of filters with bad photometry for BPZ"
			tablehead="#Bad" verbLevel="29" ucd="meta.code.qual" note="2"/>
		<column name="NfaintFilters" type="integer" required="True"
			description="Number of filters with faint photometry for BPZ"
			tablehead="#Faint" verbLevel="29" ucd="meta.code.qual" note="2"/>
		<column name="goodFilters" type="text" verbLevel="25"
			description="Filters with faint photometry for BPZ" tablehead="Good"
			note="3"/>
		<column name="badFilters" type="text" verbLevel="25"
			description="Filters with bad photometry for BPZ" tablehead="Bad"
			note="3"/>
		<column name="faintFilters" type="text" verbLevel="25"
			description="Filters with faint photometry for BPZ" tablehead="Faint"
			note="3"/>
		<column name="fieldKey" id="fieldKey" description="CARS field designation"
			ucd="meta.id" type="text" tablehead="Field"/>

		<meta name="note" tag="1"><![CDATA[
			The fluxes and related quantities are the result of a statistical
			analysis that may yield three different results, viz.,

			* invalid measurement: value and error are both NULL
			* null detection (i.e., we are sure that nothing has been detected down to
				the local limit): value 99, error NULL
			* object detection: value anything but 99, valid error

			This means that something like::

				SELECT sum(magisoU)/count(*) FROM cars.srccat WHERE <whatever>

			to estimate a mean magnitude will, in all likelihood, *not* give the correct
			result since values of 99 will be present in the sum.

			Instead, use something like::

				SELECT sum(magisoU)/count(*) FROM cars.srccat
					WHERE magisoErrU IS NOT NULL
					AND <whatever>
		]]>
		</meta>

		<meta name="note" tag="2">
			The relation between NgoodFilters/NBPZ_GOODFILT,
			NbadFilters/NBPZ_BADFILT, NfaintFilters/NBPZ_LIMFILT is
			as follows:

			The number of filters in which an object has 'reliable photometry' is
			given NgoodFilters.  Reliable photometry means
			magnitude errors &lt; 1 mag and an object brighter than the
			limiting magnitude.
			
			NbadFilters is the number of filters in which an object has formal
			magnitude errors of 1 mag or larger.
			
			NfaintFilters is the number of filters
			in which an object is fainter than the formal limiting magnitude.
			
			If an object would fall into badFilters AND faintFilters it is listed
			under badFilters. Magnitude errors refer to isophotal magnitudes; the
			adopted limiting magnitude is defined in Sect. 2 of
			2009A&amp;A...493.1197E.
		</meta>

		<meta name="note" tag="3"><![CDATA[
			For the meaning of good, bad, and faint, see
			`note 2 </__system__/dc_tables/show/tableinfo/cars.srccat#note-2>`_.

			The goodFilters/BPZ_GOODFILT, badFilters/BPZ_BADFILT,
			and failtFilters/BPZ_LIMFILT columns contain a binary encoding
			of the filters:

			======  ===  =====
			Filter  Bit  Value
			======  ===  =====
				u*     0    1
				g'     1    2
				r'     2    4
				i'     3    8
				z'     4    16
			======  ===  =====

			The columns contain the sum of the values of the filters fulfilling the
			corresponding criteria.]]>
		</meta>

		<meta name="note" tag="4">
			For non-detected objects in a filter magnitude limit columns might
			contain an invalid value NaN (NULL in the database).
		</meta>
	</table>

	<procDef id="parseFilter">
		<setup>
			<par key="enc" late="True"/>
			<par key="dest"/>
		</setup>
		<code><![CDATA[
			enc = int(enc)
			res = []
			mask = 1
			for name in 'ugriz':
				if enc&mask:
					res.append(name)
				mask <<= 1
			vars[dest] = "".join(res)]]>
		</code>
	</procDef>
	<procDef type="sourceFields" id="insertFieldKey">
		<code>
			return {"fieldKey": os.path.basename(sourceToken).split("_")[0],
				"fieldBandKey": os.path.basename(sourceToken).split(".")[0]}
		</code>
	</procDef>

	<data id="import_srccat" auto="False">
		<sources pattern="data/catalogues/*.final.asc.gz"
			recurse="True"/>
		<reGrammar gunzip="True" topIgnoredLines="50">
			<sourceFields procDef="insertFieldKey"/>
			<names>SeqNr, carsid, raj2000, dej2000, xpos, ypos,
				magAuto, magErrAuto, magisoU, magisoErrU, magisoG,
				magisoErrG, magisoR, magisoErrR, magisoI, magisoErrI,
				magisoZ, magisoErrZ, magaprU, magaprErrU, magaprG,
				magaprErrG, magaprR, magaprErrR, magaprI, magaprErrI,
				magaprZ, magaprErrZ, maglimU, maglimG, maglimR,
				maglimI, maglimZ, fwhm, fluxRadius, majorAxis,
				minorAxis, theta, sgclass, mask, extflags,
				z_b, t_b, odds, NgoodFilters, NbadFilters,
				NfaintFilters, goodFilters, badFilters, faintFilters
			</names>
		</reGrammar>
		<rowmaker id="make_srccat" idmaps="*">
			<!-- Null value policy: -99 is a null value for fluxes and such.
			99 is a non-detection in magnitude and thus a *valid* value -->
			<apply procDef="parseFilter" name="parseGoodFilter">
				<bind key="enc">vars["goodFilters"]</bind>
				<bind key="dest">"goodFilters"</bind>
			</apply>
			<apply procDef="parseFilter" name="parseBadfilt">
				<bind key="enc">vars["badFilters"]</bind>
				<bind key="dest">"badFilters"</bind>
			</apply>
			<apply procDef="parseFilter" name="parseLimfilt">
				<bind key="enc">vars["faintFilters"]</bind>
				<bind key="dest">"faintFilters"</bind>
			</apply>
			<apply name="makeNulls">
				<setup>
					<par key="nullableColumns">["magAuto", "magErrAuto", "magisoU",
							"magisoErrU", "magaprU", "magaprErrU", "maglimU",
							"magisoG", "magisoErrG", "magaprG", "magaprErrG",
							"maglimG", "magisoR", "magisoErrR", "magaprR", "magaprErrR",
							"maglimR", "magisoI", "magisoErrI", "magaprI", "magaprErrI",
							"maglimI", "magisoZ", "magisoErrZ", "magaprZ", "magaprErrZ",
							"maglimZ",]</par>
				</setup>
				<code>
					for c in nullableColumns:
						if vars[c]=='-99.0000' or vars[c]=='99.0000':
							vars[c] = None
				</code>
			</apply>
			<map dest="mask">bool(int(@mask))</map>
		</rowmaker>
		<make table="srccat" rowmaker="make_srccat"/>
	</data>

	<table id="images" onDisk="True" mixin="//siap#pgs" adql="True">
		<meta name="description" format="rst">
			Metadata for co-added CFHTLS archive images
			used for producing the CARS source list (cars.srccat).
		</meta>
		<mixin
			calibLevel="3"
			collectionName="'CARS'"
			facilityName="'CFHT'"
			sResolution="0.5"
			>//obscore#publishSIAP</mixin>
		<column original="fieldKey"/>
		<column name="fieldBandKey" id="fieldBandKey" type="text" tablehead="Field/Band"
			ucd="meta.id" description="CARS field designation plus band"/>
		<column name="ncombine" type="integer" required="True"
			description="Number of individual frames in this stack"
			tablehead="Combined" ucd="meta.number"
			verbLevel="15"/>
		<column description="Total Exposure Time" name="texptime"
			tablehead="Tot. Exposure" ucd="time.duration;obs.exposure" unit="s"
			verbLevel="15"/>
		<column description="AB Magnitude Zeropoint" name="magzp"
			tablehead="Mag. ZP" ucd="phot.mag;arith.zp" unit="mag"
			verbLevel="25"/>
		<column description="Estimated Magnitude Zeropoint Error" name="magzperr"
			tablehead="Err. Mag. ZP" ucd="stat.error;phot.mag;arith.zp"
			unit="mag"/>
		<column description="Measured image seeing" name="seeing"
			tablehead="Seeing" ucd="instr.obsty.seeing" unit="arcsec"
			verbLevel="20"/>
		<column description="Measured image Seeing error" name="seeingerr"
			tablehead="Err. Seeing" ucd="stat.error;instr.obsty.seeing"
			unit="arcsec" verbLevel="25"/>
	</table>

	<coverage>
		<updater sourceTable="images" mocOrder="4"/>
		<spatial>3/276,297 4/669-671,692-693,712,714,1093,1108,1110,1166-1167,1200,1202,2287,2298-2299,2302</spatial>
		<spectral>1.65537e-19 6.01953e-19</spectral>
	</coverage>



	<data id="import_images" auto="False">
		<sources recurse="True" pattern="data/images/*.cut.fits"/>
		<fitsProdGrammar qnd="True">
			<sourceFields procDef="insertFieldKey"/>
			<rowfilter procDef="//products#define">
				<bind key="table">"cars.images"</bind>
			</rowfilter>
		</fitsProdGrammar>

		<rowmaker id="make_images">
			<apply procDef="//siap#computePGS"/>
			<apply procDef="//siap#setMeta">
				<bind key="title">"CARS-"+vars["parser_"].sourceToken.split("/")[-2]</bind>
				<bind key="instrument">"CFHT 3.6m"</bind>
					<!-- The images are really combinations of several exposures -->
				<bind key="dateObs">None</bind>
				<bind key="bandpassId">vars["FILTER"].strip()</bind>
			</apply>
			<apply procDef="//siap#getBandFromFilter"/>
			<idmaps>fieldBandKey,fieldKey</idmaps>
			<simplemaps>texptime:TEXPTIME,
				ncombine:NCOMBINE,
				magzp:MAGZP,
				magzperr:MAGZPERR,
				seeing:SEEING,
				seeingerr:SEEINERR
			</simplemaps>
		</rowmaker>

		<make table="images" rowmaker="make_images"/>
	</data>

	<table id="weights" onDisk="True" mixin="//products#table">
		<column original="fieldBandKey"/>
	</table>

	<data id="import_weights" auto="False">
		<sources pattern="data/images/*.cut.weight.fits.gz" recurse="True"/>
		<customGrammar module="res/opaquegrammar">
			<sourceFields procDef="insertFieldKey"/>
			<rowfilter procDef="//products#define">
				<bind key="table">"cars.weights"</bind>
				<bind key="mime">"application/octet-stream"</bind>
			</rowfilter>
		</customGrammar>
		<make table="weights"/>
	</data>

	<STREAM id="regions_items">
		<doc>Common configuration for regions, masks, and catalogs</doc>
		<onDisk>True</onDisk>
		<mixin>//products#table</mixin>
		<column original="fieldKey"/>
	</STREAM>

	<table id="regions">
		<FEED source="regions_items"/>
		<meta name="description" format="rst">
	Metadata for CARS region files.

	An ASCII ds9 region file containing saoimage/ds9 polygon entries
	(in pixel coordinates). They represent the CARS image masks described
	in Sect. 3.4 of 2009A&amp;A...493.1197E. Polygons originating from different
	masking procedures are encoded with different colours (look at
	expressions such as '# color=yellow' at the end of each line).
	We offer this file so that interested users can construct own masks
	containing only parts of our global MASK key. (typical gzipped file size per field: 1MB)

	The colour coding in these files is as follows:

	- yellow:	masks from our object density analysis and stars identified
	  in the USNO-A2 catalogue
	- green:	 stars identified in the GSC1 and GSC2 cataogues
	- magenta: masks around extended haloes of bright stars
	- red:		 masks from our asteroid track analysis

		</meta>
	</table>

	<rowmaker id="genericProdimport" idmaps="*">
		<FEED source="//products#mapUserItems"/>
	</rowmaker>

	<data id="import_regions" auto="False">
		<sources pattern="data/catalogues/*.final.reg.gz" recurse="True"/>
		<customGrammar module="res/opaquegrammar">
			<sourceFields procDef="insertFieldKey"/>
			<rowfilter procDef="//products#define">
				<bind key="table">"cars.regions"</bind>
				<bind key="mime">"application/octet-stream"</bind>
			</rowfilter>
		</customGrammar>
		<make table="regions" rowmaker="genericProdimport"/>
	</data>

	<table id="truecol">
		<FEED source="regions_items"/>
		<meta name="description" format="rst">
	Metadata for low resolution true colour i', r' and g' JPG images of the CARS
	fields.  These images allow a quick look at the data but they are probably
	not suitable for high-quality poster print-outs.	(typical file size per
	field: 10MB)
		</meta>
	</table>

	<data id="import_truecol" auto="False">
		<sources pattern="data/catalogues/*.truecol.jpg" recurse="True"/>
		<customGrammar module="res/opaquegrammar">
			<sourceFields procDef="insertFieldKey"/>
			<rowfilter procDef="//products#define">
				<bind key="table">"cars.truecol"</bind>
				<bind key="mime">"image/jpeg"</bind>
			</rowfilter>
		</customGrammar>
		<make table="truecol" rowmaker="genericProdimport"/>
	</data>

	<table id="masks">
		<FEED source="regions_items"/>
		<meta name="description" format="rst"><![CDATA[
	Metadata for low resolution JPG images showing object positions and object
	masks overlayed on the CARS i'-band image. It allows a quick-look at
	field-layout and object positions.	(typical file size per field: 3MB)

	The colour coding in this is:

	- green:	areas excluded from scientific analysis by our MASK key
	- yellow: stellar sources (SExtractor CLASS_STAR >=0.95)
	- red: galaxies with high-confidence photo-z estimates:
	  (SExtractor CLASS_STAR < 0.95; reliable object photometry
	  in five bands; BPZ ODDS quality parameter > 0.9);
	  see also Sect. 4 of 2009A&A...493.1197E.
	- blue:	 all other galaxies (SExtractor CLASS_STAR < 0.95)
	  ]]></meta>
	</table>

	<data id="import_masks" auto="False">
		<sources pattern="data/catalogues/*.masks.obj.jpg" recurse="True"/>
		<customGrammar module="res/opaquegrammar">
			<sourceFields procDef="insertFieldKey"/>
			<rowfilter procDef="//products#define">
				<bind key="table">"cars.masks"</bind>
				<bind key="mime">"image/jpeg"</bind>
			</rowfilter>
		</customGrammar>
		<make table="masks" rowmaker="genericProdimport"/>
	</data>

	<table id="cats">
		<FEED source="regions_items"/>
		<meta name="description" format="rst">
	Metadata for CARS
	object catalogues in FITS binary format; beware that they are very
	big and cumbersome to handle if you do not have efficient tools to
	work on big FITS binary tables!	(typical file size per field: 0.5GB)
		</meta>
	</table>

	<data id="import_cats" auto="False">
		<meta name="description">Original products of the CARS survey.</meta>
		<sources pattern="data/catalogues/*.final.cat" recurse="True"/>
		<customGrammar module="res/opaquegrammar">
			<sourceFields procDef="insertFieldKey"/>
			<rowfilter procDef="//products#define">
				<bind key="table">"cars.cats"</bind>
				<bind key="mime">"application/octet-stream"</bind>
			</rowfilter>
		</customGrammar>
		<make table="cats" rowmaker="genericProdimport"/>
	</data>

	<table id="products" onDisk="True" namePath="images">
		<meta name="description">
			A view combining most of the CARS-related products.
		</meta>
		<LOOP>
			<listItems>
				fieldKey bandpassId ncombine texptime magzp magzperr
				seeing seeingerr accref accsize
			</listItems>
			<events>
				<column original="\item"/>
			</events>
		</LOOP>
		<column name="weightlink" type="text"
			description="Weight FITS" tablehead="Weights"
			displayHint="type=product"/>
		<column name="weightsize" type="integer" required="True"
			description="Size of weight FITS" tablehead="Size(Weights)"
			displayHint="type=size"/>
		<column name="regionlink" type="text" tablehead="DS9 masks"
			description="Image masks as DS9 regions"
			displayHint="type=product,nopreview=True"/>
		<column name="regionsize" type="integer" required="True"
			tablehead="Size(masks)"
			description="Size of DS9 region file" displayHint="type=size"/>
		<column name="truecollink" type="text"
			description="Preview color image made of i,r, and g band images"
			tablehead="Preview" displayHint="type=product"/>
		<column name="masklink" type="text" tablehead="JPEG masks"
			description="Mask preview as JPEG" displayHint="type=product,width=150"/>
		<column name="catlink" type="text" tablehead="Object catalogue"
			description="FITS catalogue of objects in this field"
			displayHint="type=product,nopreview=True"/>
		<column name="catsize" type="integer" required="True"
			tablehead="Size(cat.)"
			description="Size of FITS catalogue of objects in this field"/>
		<viewStatement>
			CREATE VIEW \curtable AS (
				SELECT im.fieldKey, im.bandpassId, im.ncombine, im.texptime, im.magzp,
					im.magzperr, im.seeing, im.seeingerr, im.accref, im.accsize,
					cars.weights.accref as weightlink,
					cars.weights.accsize as weightsize,
					cars.regions.accref as regionlink,
					cars.regions.accsize as regionsize,
					cars.truecol.accref as truecollink,
					cars.masks.accref as masklink,
					cars.cats.accref as catlink,
					cars.cats.accsize as catsize
				FROM cars.images AS im
				JOIN cars.weights USING (fieldBandKey)
				JOIN cars.regions USING (fieldKey)
				JOIN cars.truecol USING (fieldKey)
				JOIN cars.masks USING (fieldKey)
				JOIN cars.cats USING (fieldKey))
		</viewStatement>
	</table>

	<data id="create_products_view" auto="False">
		<make table="products"/>
	</data>

	<data id="create_prereqs" auto="False">
		<!-- all tables necessary for the products view -->
		<recreateAfter>import_images</recreateAfter>
		<recreateAfter>import_weights</recreateAfter>
		<recreateAfter>import_regions</recreateAfter>
		<recreateAfter>import_truecol</recreateAfter>
		<recreateAfter>import_masks</recreateAfter>
		<recreateAfter>import_cats</recreateAfter>
	</data>

	<data id="import_products">
	<!-- a "virtual data" that just causes all products-related tables to be
	imported; this is the one you want to import products	-->
		<recreateAfter>create_prereqs</recreateAfter>
		<recreateAfter>create_products_view</recreateAfter>
	</data>

	<dbCore id="prodCore" queriedTable="products">
		<condDesc>
			<inputKey original="fieldKey" showItems="7" multiplicity="multiple">
				<values fromdb="fieldKey from cars.products"/>
			</inputKey>
		</condDesc>
		<condDesc>
			<inputKey original="bandpassId" multiplicity="multiple">
				<values fromdb="bandpassId from cars.products"/>
			</inputKey>
		</condDesc>
	</dbCore>

	<service id="dl" allowed="dlget,dlmeta,static">
		<meta name="title">Datalink service</meta>
		<property key="staticData">data</property>
		<datalinkCore>
			<descriptorGenerator procDef="//soda#fits_genDesc">
				<bind name="accrefPrefix">"cars/data"</bind>
			</descriptorGenerator>
			<metaMaker name="addRelated" semantics="#source-list">
				<setup>
					<code>
					_linksByExt = {
						"truecol.jpg": {
							"contentType": "image/jpeg", "semantics": "#preview-image",
							"description": "Color preview"},
						"obj.jpg": {
							"contentType": "image/jpeg", "semantics": "#preview-image",
							"description": "Preview of the object mask"},
						"reg.gz": {
							"contentType": "application/octet-stream",
							"description": "DS9 masks for unusable regions",
							"semantics": "#auxiliary"},
						"asc.gz": {
							"contentType": "text/plain", "semantics": "#source-list",
							"description": "Extracted sources."},
					}
					</code>
				</setup>
				<code>
					root = "cars/q/dl/static/"
					stem = descriptor.accref.split("/")[-1][:-15]
					dir = descriptor.accref.split("/")[-2]
					catDir = os.path.join(base.getConfig("inputsDir"),
						"cars/data/catalogues", dir)
					yield descriptor.makeLink(makeAbsoluteURL(root+
						"images/%s/%s.swarp.cut.weight.fits.gz"%(dir, stem)),
						contentType="application/fits", description="Weights",
						semantics="#weight")

					for fName in os.listdir(catDir):
						try:
							yield descriptor.makeLink(makeAbsoluteURL(root+
								"catalogues/%s/%s"%(dir, fName)),
								**_linksByExt[".".join(fName.split(".")[-2:])])
						except KeyError:
							# unknown extension, ignore
							pass
				</code>
			</metaMaker>

			<metaMaker semantics="#proc">
				<code>
					yield descriptor.makeLink("http://dc.g-vo.org/tap",
						description="TAP service with this data in obscore")
				</code>
			</metaMaker>

			<FEED source="//soda#fits_standardDLFuncs"
				stcs='PositionInterval ICRS "RA_MIN" "DEC_MIN" "RA_MAX" "DEC_MAX"'
				accrefStart="cars/data/images"/>

		</datalinkCore>
	</service>

	<service id="prod" core="prodCore">
		<meta name="title">CARS product query</meta>
		<meta name="_related" title="CARS source catalogue">/cars/q/cat/form</meta>
		<meta name="_intro" format="rst">A simple search interface to the
		images used to generate the
		\RSTservicelink{/cars/q/cat/form}{CARS source catalogue}.</meta>

		<meta name="_bottominfo" format="raw"><![CDATA[
			<p>Current data coverage:</p>
			<a href="\internallink{cars/q/cat/static/fields.png}">
			<img src="\internallink{cars/q/cat/static/fields-inline.png}"/>
			</a>]]>
		</meta>

		<outputTable namePath="products">
			<outputField original="fieldKey" sets="ALL"/>
			<outputField original="bandpassId" sets="ALL"/>
			<outputField original="texptime" sets="ALL"/>
			<outputField original="ncombine" sets="2,3"/>
			<outputField original="magzp" sets="2,3"/>
			<outputField original="seeing" sets="2,3"/>
			<outputField original="truecollink" sets="1,2,3"/>
			<outputField name="image" original="accref" sets="2,3"
				tablehead="Image data"
				select="array[accref, accsize::TEXT]" type="raw">
				<formatter>
					return T.a(href=makeProductLink(data[0]))[
						"Download %s"%utils.formatSize(int(data[1]))]
				</formatter>
			</outputField>
			<outputField name="weight" original="weightlink" sets="3"
				select="array[weightlink, weightsize::TEXT]" type="raw">
				<formatter>
					return T.a(href=makeProductLink(data[0]))[
						"Download %s"%utils.formatSize(int(data[1]))]
				</formatter>
			</outputField>
			<outputField original="masklink" sets="2,3"/>
			<outputField name="catalogue" original="catlink" sets="ALL"
				select="array[catlink, catsize::TEXT,fieldKey]" type="raw">
				<formatter>
					return T.a(href=makeProductLink(data[0]))[
						"%s objects (%s)"%(data[2], utils.formatSize(int(data[1])))]
				</formatter>
			</outputField>
		</outputTable>
		<inputKey name="_SET" type="text" tablehead="Output with">
			<values>
				<option title="Preview">1</option>
				<option title="Source Image">2</option>
				<option title="All Products">3</option>
			</values>1</inputKey>
	</service>

	<scsCore id="catCore" queriedTable="srccat">
		<condDesc original="//scs#humanInput"/>
		<condDesc original="//scs#protoInput"/>
		<condDesc buildFrom="magAuto"/>
		<condDesc buildFrom="sgclass"/>
	</scsCore>

	<service id="cat" core="catCore" allowed="form,static,scs.xml">
		<property name="staticData">static</property>
		<publish render="form" sets="local,ivo_managed"/>
		<publish render="scs.xml" sets="ivo_managed"/>
		<meta name="title">CARS source catalogue query</meta>
		<meta name="shortName">CARS_sources</meta>
		<meta name="_intro" format="rst">A simple search interface for sources extracted by
		CARS.  This data is also exposed to
		\RSTservicelink{__system__/adql/query/form}{ADQL} to allow more advanced
		queries.</meta>
		<meta name="_related" title="CARS product query">/cars/q/prod/form</meta>
		<meta name="_bottominfo" format="raw"><![CDATA[
			<p>Current data coverage:</p>
			<a href="\internallink{cars/q/cat/static/fields.png}">
			<img src="\internallink{cars/q/cat/static/fields-inline.png}"/>
			</a>
			<script language="text/html" id="aladinScriptWindow">
				<div>
					<p>Aladin script:<br/><code>
					load 'http://dc.zah.uni-heidelberg.de/cars/q/cutout/siap.xml?POS=$RA,$DEC&amp;SIZE=0.05&amp;cutoutSize=0.05';<br/>
						load 'http://dc.zah.uni-heidelberg.de/cars/q/cat/scs.xml?RA=$RA&amp;DEC=$DEC&amp;SR=0.05';
						</code>
					</p>
				</div>
			</script>
			]]></meta>
		<meta name="testQuery">
			<meta name="ra">36</meta>
			<meta name="dec">-6</meta>
			<meta name="sr">0.01</meta>
		</meta>
		<outputTable namePath="srccat">
			<LOOP>
				<listItems>
					_r raj2000 dej2000 sgclass magAuto magErrAuto magaprU magaprG
					magaprR magaprI magaprZ fwhm
				</listItems>
				<events>
					<outputField original="\item"/>
				</events>
			</LOOP>
			<outputField original="fieldKey" displayHint="noxml=true">
				<formatter>
					yield T.a(href="/cars/q/prod/form?__nevow_form__=genForm"
						"&amp;fieldKey=%s&amp;_SET=3"%data)[
						"Source products"]
				</formatter>
			</outputField>
			<outputField name="cutout" select="array[raj2000, dej2000]"
					tablehead="Aladin"
					description="View the generating images in Aladin.  The applet version of aladin might run out of memory for these links.  The script links gives you an Aladin script for cut'n'paste."
					displayHint="noxml=true">
				<formatter><![CDATA[
					aladinPrefix = ("http://aladin.u-strasbg.fr/java/nph-aladin.pl"
						"?-script=")
					rd = base.caches.getRD("cars/q")
					imgURL = rd.getById("cutout").getURL(
						"siap.xml", absolute=True)+(
						"POS=%s,%s&SIZE=0.05&cutoutSize=0.05")%(data[0], data[1])
					catURL = rd.getById("cat").getURL(
						"scs.xml", absolute=True)+(
						"RA=%s&DEC=%s&SR=0.05"%(data[0], data[1]))
					script = "load '%s';load '%s';"%(imgURL, catURL)
					yield T.a(href=aladinPrefix+urllib.parse.quote(script))[
						"[Show]"]
					yield " "
					yield T.a(onclick="popupInnerWindow($(renderTemplate(%s)))"%
						("'aladinScriptWindow', {'RA': %s, 'DEC': %s}"%(data[0], data[1]))
						)["[Script]"]
				]]></formatter>
			</outputField>
		</outputTable>
	</service>

	<siapCutoutCore id="cutout_core" queriedTable="images">
		<condDesc original="//siap#protoInput"/>
		<condDesc original="//siap#humanInput"/>
	</siapCutoutCore>

	<service id="cutout" core="cutout_core" allowed="form,siap.xml">
		<meta name="title">CARS image cutouts</meta>
		<meta name="shortName">CARS_cutout</meta>
		<inputKey name="cutoutSize" unit="deg" widgetFactory="Hidden"
			>0.01</inputKey>
		<meta name="_related" title="CARS product query">/cars/q/prod/form</meta>
		<publish render="form" sets="local,ivo_managed"/>
		<publish render="siap.xml" sets="ivo_managed"/>
		<meta name="sia.type">Cutout</meta>
		<meta name="testQuery.pos.ra">36</meta>
		<meta name="testQuery.pos.dec">-6</meta>
		<meta name="testQuery.size.ra">0.05</meta>
		<meta name="testQuery.size.dec">0.05</meta>
	</service>

	<regSuite id="dachstest">
		<regTest title="rdinfo displays tables and services with some infos">
			<url>/browse/cars/q</url>
			<code><![CDATA[
				self.assertHasStrings("resource \'CARS survey",
					'/cars/q/cutout/info">CARS image cutouts',
					'<div class="lidescription"><p>Metadata for CARS')
			]]></code>
		</regTest>

		<regTest title="JPEG preview looks like a jpeg" tags="bigserver">
			<url preview="True">/getproduct/cars/data/catalogues/W1p1p1/
				W1p1p1_izrgu.V1.7A.masks.obj.jpg</url>
			<code>
				self.assertHasStrings("JFIF", "22222222222222222222222")
				# it would be good if we could remove the cache generated
				# so we don't just test locating the preview cache.  but
				# that'd need some special trick as that's server side
				# But just in case we're running on the box tested:
				try:
					os.unlink(os.path.join(base.getConfig("web", "previewCache"),
						getFlatName(re.sub(".*getproduct/", "", self.url.httpURL))))
				except os.error:
					pass
			</code>
		</regTest>
	</regSuite>

	<regSuite id="servicetest">
		<regTest title="CARS product search returns some data" tags="bigserver">
			<url parSet="form" fieldKey="W1p1p1" _SET="3">prod/form</url>
			<code><![CDATA[
				self.assertHasStrings("<td>W1p1p1",
					'title="Mask preview as JPEG"',
					"/cars/data/images/W1p1p1/W1p1p1_g.V1.7A.swarp.cut.weight.fits.gz",
					"Download 54.0MiB",
					'onmouseover="insertPreview(this, null)"',
					"W1p1p1_izrgu.V1.7A.masks.obj.jpg")

			]]></code>
		</regTest>

		<regTest title="CARS Source cat query returns expected results"
			tags="bigserver">
			<url RA="34.00953702" DEC="-6.58417263" SR="0.0001">cat/scs.xml</url>
			<code>
				row = self.getFirstVOTableRow()
				self.assertAlmostEqual(row["magaprG"], 24.427299499512)
				self.assertEqual(row["carsid"], 'W1m0p1:154')
				self.assertEqual(row["SeqNr"], 154)
				self.assertEqual(row["magisoR"], None)
				self.assertEqual(row["magaprErrR"], None)
			</code>
		</regTest>

		<regTest title="CARS Source web query has fancy additional fields"
			tags="bigserver">
			<url parSet="form" hscs_pos="36 -6" hscs_sr="1"
				>cat/form</url>
			<code>
				self.assertHasStrings(
					"$(renderTemplate('aladinScriptWindow', {'RA': 35.99848722, 'DEC': -5.993341385})))"
					)
			</code>
		</regTest>

		<regTest title="CARS datalink looks about ok" tags="bigserver">
			<url ID="ivo://org.gavo.dc/~?cars/data/images/W1m0p1/W1m0p1_g.V1.7A.swarp.cut.fits"
				>dl/dlmeta</url>
			<code>
				self.assertHasStrings("#weight",
					"cars/q/dl/static/"
						"images/W1m0p1/W1m0p1_g.V1.7A.swarp.cut.weight.fits.gz",
					"#preview-image",
					"/cars/q/dl/static/"
						"catalogues/W1m0p1/W1m0p1_irg.V1.7A.truecol.jpg",
					'value="34.5', "-6.066", "0.76")
			</code>
		</regTest>

		<regTest title="CARS SODA works" tags="bigserver">
			<url ID="ivo://org.gavo.dc/~?cars/data/images/W1m0p1/W1m0p1_g.V1.7A.swarp.cut.fits"
				CIRCLE="34.5 -6.0 0.001"
				>dl/dlget</url>
			<code>
				self.assertHasStrings(
					"NAXIS1  =                   39",
					b"\\x3d\\x40\\xb4\\xc1\\xbc\\x5b")
			</code>
		</regTest>
	</regSuite>
</resource>
