2024-01-25T14:24:17Z 100 DASCH in the VO This is a re-publication of the metadata from StarGlass, an information service operated by Harvard College Observatory (HCO) which primarily indexes the holdings of the HCO Astronomical Photographic Glass Plate Collection. This collection is the world's largest archive of astronomical photographic glass plates, consisting of more than 550,000 images documenting the entire sky over a century-long time baseline. Approximately 430,000 plates in the collection, representing the subset of sky images deemed suitable for astrometric and photometric calibration, have been digitized at high resolution by the DASCH project. StarGlass makes available both photographs and calibrated scientific data when available. This resource re-publishes the metadata in separate tables for “narrow” and “wide” (extremely wide-field) plates. The narrow plates are also published in an obscore table and as a SIAP2 service. history-of-astronomy astrophotography Harvard College Observatory Eastman Aero-Ektar K-24 Lens on a K-19 Barrel Formerly KG (1950) Damons North Red Cooke Short Focus Damons South Blue Damons South Yellow 4-inch Ross Lundin 24-33 in Jewett Schmidt Cerro Tololo 4 meter 4-inch Voightlander Lens 8-inch Brashear Lens 4-inch Cooke (1-327) Eastman Aero-Ektar K-24 Lens on a K-19 Barrel Formerly KE (1951) 3-inch Ross 1 in Cook Lens #832 Series renamed from ac-a 24-inch Clark Reflector Air Force Camera 10-inch Metcalf Triplet 13-inch Boyden Refractor (spectrum plates) 3 inch Perkin-Zeiss Lens Lowel 40 inch reflector 1.5 inch Ross (short focus) Damons South Yellow 4-inch Ross Lundin 24-33 in Jewett Schmidt Cerro Tololo 4 meter 4-inch Voightlander Lens 8-inch Brashear Lens 4-inch Cooke (1-327) Eastman Aero-Ektar K-24 Lens on a K-19 Barrel Formerly KE (1951) 3-inch Ross 1 in Cook Lens #832 Series renamed from ac-a 24-inch Clark Reflector Air Force Camera 10-inch Metcalf Triplet 13-inch Boyden Refractor (spectrum plates) 3 inch Perkin-Zeiss Lens Lowel 40 inch reflector 1.5 inch Ross (short focus) 8-inch Draper Doublet Logbook Only. Pages without plates. 61-inch Wyeth Reflector Roe 6-inch 2.6-inch Zeiss-Tessar Damons North Yellow Damons South Red 16-inch Boller & Chivens New Cooke Lens 5-in Voightlander Transit Photometer 11-inch Draper Refractor 12-inch Metcalf Doublet 8-inch Bache Doublet Patrol cameras 3-inch Ross Fecker 24-inch Bruce Doublet KE Camera with Installed Rough Focus 3-inch Darlot (Series renamed from j) 8-inch Ross Lundin 32-36 inch BakerSchmidt 10 1/2 inch round Armagh-Dunsink-Harvard K-19 Air Force Camera 16-inch Metcalf Doublet (Refigured after 3500) Eastman Aero-Ektar K-24 Lens on a K-19 Barrel 3 inch Ross-Tessar Lens 7.5-inch Cooke/Clark Refractor at Maria Mitchell Observatory 1.5-inch Ross-Xpress 2.5-inch Ross Portrait 2.5-inch Ross Portrait Lens 60 inch Common 2.5 inch Voigtlander (Little Bache or "Bachito") 3-inch Elmer Ross 4-inch Cooke Lens 2.8-inch Kodak Aero-Ektar 1-inch 200 inch Hale Telescope 60-inch Rockefeller Reflector 2.5 inch Cooke Lens Asiago Observatory 92/67 cm Schmidt Damons North Blue 1.5-inch Cooke Lenses Palomar Sky Survey (POSS) 1.5 in Cooke "Long Focus" Miscellaneous test plates Misc Basement Various Meteor Cameras 13-inch Boyden Refractor Pole Star Recorder YSO Double Astrograph Harvard-Smithsonian Center for Astrophysics 2012IAUS..285...29G Research Archive Optical Narrow and and wide plates from DASCH have the same schema; this stream contains the common elements. //siap2#pgs _associatedDatalinkService.serviceId: dl _associatedDatalinkService.idColumn: obs_publisher_did //obscore#publishObscoreLikeThis table holds metadata for the parts of DASCH counting as targeted observations (plate scale below 400 arcsec/mm. “Patrol” and “Meteor” plates in DASCH nomenclature are found in the wide_plates table. This table holds metadata for the “Patrol” and “Meteor” plates from DASCH, i.e., very wide-field observations presumably not useful in global discovery. These data products are therefore not re-published through obscore. For the “narrow” plates, see the narrow_plates table. This table is a union of the narrow_plates and wide_plates tables. For non-historic use, you probably get better results looking at narrow_plates. CREATE VIEW \qName AS ( SELECT \colNames FROM dasch.narrow_plates UNION ALL SELECT \colNames FROM dasch.wide_plates) def condget(val, *keys): for key in keys: if isinstance(key, int): val = val[key] else: val = val.get(key) if val is None: break return val def to_wcs_keys(exposure): """returns a dasch exposure record with proper wcs keys. """ key_map = { "ctr_dec": "CRVAL2", "ctr_ra": "CRVAL1", "delta_dec_x": "CD2_1", "delta_dec_y": "CD2_2", "delta_ra_x": "CD1_1", "delta_ra_y": "CD1_2", "crpix1": "CRPIX1", "crpix2": "CRPIX2", "naxis1": "NAXIS1", "naxis2": "NAXIS2", } return dict((key_map.get(k, k), v) for k, v in exposure.items()) # we sort into wide and narrow by instrument name rather # than plate scale because we don't have the plate scale # for may plates. The following list is the result of # looking at the plate scales of the initial import in May 2024, # See blog post from May 2024 WIDE_INSTRUMENTS = { "Eastman Aero-Ektar K-24 Lens on a K-19 Barrel Formerly KG (1950)", "Logbook Only. Pages without plates.", "Roe 6-inch", "1.5 inch Ross (short focus)", "Patrol cameras", "1.5-inch Ross-Xpress", "2.8-inch Kodak Aero-Ektar", "KE Camera with Installed Rough Focus", "Eastman Aero-Ektar K-24 Lens on a K-19 Barrel", "Eastman Aero-Ektar K-24 Lens on a K-19 Barrel Formerly KE (1951)", "3 inch Perkin-Zeiss Lens", "3 inch Ross-Tessar Lens", "2.6-inch Zeiss-Tessar", "Air Force Camera", "K-19 Air Force Camera", "1.5 in Cooke \"Long Focus\"", "1 in Cook Lens #832 Series renamed from ac-a", "1-inch", "1.5-inch Cooke Lenses", "2.5 inch Cooke Lens", "2.5-inch Ross Portrait Lens", "Damons South Yellow", "Damons South Red", "Damons North Red", "Damons North Blue", "Damons North Yellow", "New Cooke Lens", "Damons South Blue", "2.5 inch Voigtlander (Little Bache or \"Bachito\")", "NULL", "3-inch Ross Fecker", "3-inch Ross", "3-inch Elmer Ross", "4-inch Ross Lundin", } # these DASCH ids need to be investigated; their metadata # records look really awful completely_broken = {'ai43984', 'ai43986'} with open(self.sourceToken, encoding="utf-8") as f: result = json.load(f) if result["num_errors"]!=0: base.ui.notifyWarning( f"{result['num_errors']} errors in {self.sourceToken}") for plateid, platedata in result["results"].items(): if plateid in completely_broken: continue metadata = platedata["data"] if metadata is None: # The CSV apparently contains a few non-existing plates. # Skip these. continue record = {"dasch_id": plateid, "status": platedata["status"], "instrument_name": metadata.get("telescope"), "plate_comment": metadata.get("plate_comment"), "comment_astronomers": metadata.get("comment_astronomers"), "plate_class": metadata.get("class"), "timestamp": condget(metadata, "catalog_exposures", 0, "datetime"), "facility_name": condget(metadata, "location", "name"), "mosaics": metadata.get("mosaics"), "NAXIS1": None, "NAXIS2": None, "NAXIS": 2, } wcs = metadata.get("exposures") if wcs and len(wcs)==1: record.update(to_wcs_keys(wcs[0])) record.update({ "CTYPE1": 'RA---TAN-SIP', "CTYPE2": 'DEC--TAN-SIP', "LONPOLE": 180., "CUNIT1": "deg", "CUNIT2": "deg"}) datalinks = {} for key in ["jacket_images", "plate_images", "jacket_thumbnail", "plate_thumbnail"]: datalinks[key] = [r["url"].split("?")[0] for r in metadata[key]] record["datalinks"] = datalinks record["accref"] = "dasch/q/"+record["dasch_id"] if record["instrument_name"] in WIDE_INSTRUMENTS: record["isWide"] = True yield ("wide", record) else: record["isWide"] = False yield ("narrow", record) yield ("products", record) baseURL = rd.getById("dl").getURL("dlmeta") "changes-per-row" @accref baseURL+"?ID="+urllib.parse.quote_plus( getStandardPubDID(@accref)) 'application/x-votable+xml;content=datalink' 10000 if @isWide: @prodtblTable = "dachs.wide_plates" else: @prodtblTable = "dachs.narrow_plates" yield row dasch_id, plate_comment, plate_class, datalinks, mosaics @exposure*60 if not @timestamp: base.ui.notifyInfo("No timestamp on DASCH record: {}". format(@dasch_id)) raise IgnoreThisRow("no timestamp") @centre = @t_min = @t_max = None return exptime = @t_exptime or 0 @centre = dateTimeToMJD(parseISODT(@timestamp)) @t_min = @centre-exptime/86400./2 @t_max = @centre+exptime/86400./2 result["s_xel1"] = result["s_xel2"] = None try: addWCS(vars, result) except Exception as msg: nullOutWCS(result) if vars.get("NAXIS1") is not None: base.ui.notifyWarning( f"Broken WCS ignored on {vars['dasch_id']} ({msg})") "Photographic emulsion; details pending" 2 if result["s_region"] else 1 @centre @instrument_name 'DASCH' (@instrument_name+" observation on " +mjdToDateTime(@centre).isoformat()) @t_exptime @t_min @t_max DASCH VO Datalink Service This service generates authorised links to access DASCH FITS files. To avoid unneccessarly large downloads, the #this link is a downsampled image. To retrieve full-resolution FITS-es, look for rows with semantics="#coderived" and local semantics full-res. The dlget service is a SODA emulation; however, the cutouts have a fixed size. "dasch/q/" "image" def addExtras(descriptor): descriptor.suppressAutoLinks = True with base.getTableConn() as conn: descriptor.extMeta = next(conn.queryToDicts( "SELECT * FROM dasch.plates" " WHERE obs_publisher_did = %(did)s", {"did": descriptor.pubDID})) # the original plan has been to use mosaics to infer # what is actually there. But the links are not # useful because they can only be used with an API # key. So byBinF is unused for now and we make # our own links. descriptor.byBinF = {} for m in descriptor.extMeta["mosaics"]: descriptor.byBinF.setdefault(m["bin_factor"], [] ).append((m["mosaic_num"], m["bin_factor"], m["flags"])) descriptor.datalinks = descriptor.extMeta["datalinks"] descriptor.prodURL = rd.getById("getprod").getURL("qp") descriptor.plateid = descriptor.extMeta["dasch_id"] try: estimatedSize = descriptor.extMeta["s_xel1" ]*descriptor.extMeta["s_xel2"]*16 except TypeError: estimatedSize = 100000000 yield descriptor.makeLink( descriptor.prodURL+"/fits-16/"+descriptor.plateid, contentType="application/fits", description=f"Plate scan downsampled by a factor of 16", contentLength=estimatedSize/16**2, localSemantics=f"downsampled-16") yield descriptor.makeLink( descriptor.prodURL+"/fits-01/"+descriptor.plateid, contentType="application/fits", description=f"Full-scale plate scan", contentLength=estimatedSize, semantics="#coderived", localSemantics=f"fullscale") if "plate_images" in descriptor.datalinks: yield descriptor.makeLink( descriptor.prodURL+"/preview/"+descriptor.plateid, contentType="image/jpeg", description=f"Thumbnail for the plate scan") if "plate_images" in descriptor.datalinks: yield descriptor.makeLink( descriptor.prodURL+"/jpeg/"+descriptor.plateid, contentType="image/jpeg", description=f"A lo-fi rendition of the plate scan") if "jacket_images" in descriptor.datalinks: yield descriptor.makeLink( descriptor.prodURL+"/jacket/"+descriptor.plateid, contentType="image/jpeg", description=f"A photo of the plate jacket") # leave a data attribute so the machinery is happy. The # actual upstream request will be in do-remote-cutout descriptor.data = [] # Regrettably, the SODA cutout can come in all kinds of crappy # ways. if args.get("CIRCLE"): pos = args["CIRCLE"].center elif args.get("POLYGON"): pos = pgsphere.SPoint(*args["POLYGON"].getCenter()) elif args.get("RA"): # we'd have to be a lot more careful here, what with stitching # lines, simple centres, and so on. But let's see if anyone # even uses this. pos = pgsphere.SPoint.fromDegrees( sum(args["RA"])/2, sum(args["DEC"])/2) else: raise ValidationError("Cannot find a position I recognise", colName="POS") ra, dec = pos.asDALI() payload = { "center_ra_deg": ra, "center_dec_deg": dec, "plate_id": descriptor.extMeta["dasch_id"], "solution_number": 0, } resp = requests.post("https://api.starglass.cfa.harvard.edu/public/dasch/dr7/cutout", json=payload) resp.raise_for_status() descriptor.data = gzip.decompress(base64.b64decode(resp.text)) return ("application/fits", descriptor.data) DASCH Product Access Yield pre-authenticated links to DASCH data products. Successful calls will redirect to the artefact in question. productpath apiTemplates = { "fits-01": "/plates/p/{plateid}/mosaic?bin_factor=01", "fits-16": "/plates/p/{plateid}/mosaic?bin_factor=16", "jacket": "/plates/p/{plateid}/download?image_type=jacket", "jpeg": "/plates/p/{plateid}/download?image_type=plate", "preview": "/plates/p/{plateid}/download?image_type=plate" "&thumbnail_ratio=16", } apiURL = "https://api.starglass.cfa.harvard.edu/full" with open(rd.getAbsPath("dasch-api-key")) as f: apiKey = f.read().strip() [^/]+)/(?P[^/]+)$", path) if not mat: raise svcs.UnknownURI( f"Invalid productpath '{path}'.", hint="Good paths have the form kind/plateid") kind, plateid = mat.groups() if kind not in apiTemplates: raise svcs.UnknownURI( f"Unknown product kind '{kind}'", hint=f"Known kinds are {apiTemplates.keys()}") apiCall = apiTemplates[kind].format(**locals()) resp = requests.get( apiURL+apiCall, headers={"x-api-key": apiKey}, allow_redirects=False) # aw god; sometimes this a JSON file with a link, sometimes # it's a redirect. Ah well. try: if resp.status_code==200: newURL = resp.json()["presigned_link"] else: newURL = resp.headers["location"] except Exception as msg: raise base.ui.logOldExc( svcs.UnknownURI( "DASCH presign failed -- does this plate exist?", hint=f"The local symptom was {msg}")) raise svcs.Found(newURL) ]]> DASCH DASCH SIAP2 SIAP2 service over the DASCH (Harvard) plates. Please note that a substantial number of plates has no calibration and hence will not be found using SIAP2 queries with positional constraints. Also, this service only returns ”narrow” plates, which here excludes plates from instruments with a field of view above 200 square degrees. sia.type: Pointed testQuery.pos.ra: 10 testQuery.pos.dec: 10 testQuery.size.ra: 1 testQuery.size.dec: 1 /tap/sync recs = dict((r["dasch_id"], r) for r in self.getVOTableRows()) # a01299 is not solved rec = recs["a01299"] self.assertEqual(rec["s_ra"], None) self.assertEqual(rec["facility_name"], "Harvard-Smithsonian Center for Astrophysics") self.assertEqual(rec["instrument_name"], "24-inch Bruce Doublet") self.assertAlmostEqual(rec["t_min"], 13247.0419999999) # a01419 is solved rec = recs["a01419"] self.assertAlmostEqual(rec["s_ra"], 107.87146626170235) self.assertAlmostEqual(rec["s_fov"], 8.602721773801411, 6) self.assertAlmostEqual(rec["t_max"], 13316.066861111029) self.assertAlmostEqual(rec["s_pixel_scale"], 0.6547661030197105) self.assertAlmostEqual(rec["t_exptime"], 840.0) self.assertAlmostEqual(rec["plate_comment"], "M.E.H.") /dasch/q/im/siap2.xml rec = self.getFirstVOTableRow() self.assertEqual( rec['obs_title'], '24-inch Bruce Doublet observation on 1895-05-03T01:29:16.799993') self.assertEqual(rec['s_xel1'], 31627) /dasch/q/dl/dlmeta bySemantics = self.datalinkBySemantics() self.assertEqual( set(bySemantics), { '#this', '#preview-image', '#preview', '#documentation', '#coderived', "#proc"}) self.assertEqual( self.getUnique(bySemantics["#coderived"])["local_semantics"], "fullscale") self.assertEqual( self.getUnique(bySemantics["#documentation"])["access_url"], EqualingRE(".*/dasch/q/getprod/qp/jacket/a01299")) self.assertEqual( self.getUnique(bySemantics["#preview"])["description"], "Thumbnail for the plate scan") self.assertEqual( self.getUnique(bySemantics["#this"])["content_length"], 390625) # this is wrong because I can't see NAXISn here. getprod/qp/junk self.assertHTTPStatus(404) self.assertHasStrings("Invalid productpath 'junk'.") getprod/qp/junk/a01299 self.assertHTTPStatus(404) self.assertHasStrings("'fits-01', 'fits-16',") # from the hint getprod/qp/fits-16/a01299 self.assertHTTPStatus(302) nextURL = regtest.getHeaderValue(self.headers, "location") self.assertTrue(nextURL.startswith("https://dasch-prod-user.s3.us-east-1.amazonaws.com/plates/a01299/a01299_mosaic_00_16r270ww.fit.fz")) im/siap2.xml row = self.getFirstVOTableRow() self.assertEqual( row["access_format"], "application/x-votable+xml;content=datalink") self.assertAlmostEqual( row["t_max"], 13316.066861111029) self.assertAlmostEqual( row["s_pixel_scale"], 0.6547661030197105) /tap/sync row = self.getFirstVOTableRow() self.assertEqual(row["t_exptime"], 960.0) self.assertAlmostEqual(row["t_max"], 12854.16455555554) self.assertAlmostEqual(row["s_pixel_scale"], 0.6553118760334123) /tap/sync row = self.getFirstVOTableRow() self.assertEqual(row["plate_comment"], "emulsion severely yellowed") self.assertAlmostEqual(row["t_min"], 34092.35763888899) # the next assertion ought be become false when upstream has a # better way of including NAXISn self.assertAlmostEqual(row["s_xel1"], None) dl/dlget self.assertHasStrings( "CRVAL1 = 129.7", "SIMPLE = T")