2024-10-31T12:55:21Z ROME/REA Timeseries Photometry Data Release 1 The ROME/REA (Robotic Observations of Microlensing Events/Reactive Event Assessment) Survey was a Key Project at Las Cumbres Observatory (hereafter LCO) which continuously monitored 20 selected fields (3.76 sq.deg.) in the Galactic Bulge throughout their seasonal visibility window over a three-year period, between March 2017 and March 2020. Observations were made in three optical passbands (SDSS g', r', i'), and LCO's multi-site telescope network enabled the survey to achieve a typical cadence of ~10hrs in i' and ~15hrs in g' and r'. In addition, intervals of higher cadence (<1hr) data were obtained during monitoring of key microlensing events within the fields. This catalog includes the full timeseries photometry for all ~8 million stars, down to a limiting magnitude of i~18mag. As a short example for how to explore this data using VO protocols, try this: * Open TOPCAT and it's VO/TAP window. Select "GAVO DC TAP" from the server selection and hit "use service". * We would like to find "unusual" time series; perhaps ones with a large range. To get a feeling for what "unusual" might mean, just fetch a few metadata rows like with a TAP query like this:: SELECT TOP 7000 * FROM rome.time_series WHERE np_i>100 AND np_g>100 * Do a plane plot of that data, and ask TOPCAT to plot ``mag_i_max-mag_i_min`` against ``mag_g_max-mag_g_min``. * When you open a table view, you can inspect the metadata for the time series when clicking on a point in the plot. * Let's inspect the actual data: In TOPCAT's main window, do Views/Activation Actions. If you check "Plot Table", TOPCAT will download and plot the time series; it will only plot the first lightcurve, though. To get all bands, at this point you have to manually fetch accref from the table and use File -> Load Table. Sorry about this. * Using datalink, you can also view three-band previews with one click on a plot point. To configure this, check "Invoke Service" in the Activation Actions, then use "Invoke now". In the dialog that pops up then, select the row with #preview and check Auto-Invoke. You will then see a light rendition of the light curves in (up to) three colours. Your desktop could then look somewhat like this: .. image:: /\rdId/sdl/static/rome-action.jpeg broad-band-photometry ccd-photometry light-curves multi-color-photometry photometry sloan-photometry stellar-photometry gravitational-microlensing variable-stars 50 Street, R.A.; Bachelet, E.; Tsapras, Y.; Hundertmark, M.P.G.; Bozza, V.; Bramich, D.M.; Cassan, A.; Dominik, M.; Figuera Jaimes, R.; Horne, K.; Mao, S.; Saha, A.; Wambsganss, J.; Zang, W. Sinistro cameras Las Cumbres Observatory Telescope Network 2024PASP..136f4501S Research Survey Optical survey archival timeseries MAXREC=1 This data release is a public product of the ROME/REA Key Project survey conducted on the robotic telescope network of the Las Cumbres Observatory (LCO). Individual images used in the data reduction are freely available for download on the LCO archive (https://archive.lco.global/). It is important to clarify that the light curves available through this data release are not optimized photometric reductions of individual objects. Researchers intending to utilize this data set for publications are kindly requested to acknowledge this source by citing the present paper (Street et al 2024 :bibcode:`2024PASP..136f4501S`) along with the original work by Tsapras et al 2019 :bibcode:`2019PASP..131l4401T`. For research work that requires enhanced photometry for specific targets within this catalog, we encourage reaching out to us directly. Collaboration opportunities are sincerely welcomed, and we look forward to supporting further inquiries and investigations. filter, filterHuman, ucd i, SDSS i', em.opt.i g, SDSS g', em.opt.v r, SDSS r', em.opt.r def get_photpoints(field, quadrant, quadrant_id): """returns the photometry points for the specified time series from the HDF5 as a numpy array. Regrettably, null value handling is a pain. We replace 0 and -9999 with NaNs. You will get back everything, which for most time series is basically only junk. Do throw away everything that does not have isnan(rec[-1]). We don't do that here because the array indexes are significant. And sorry for a NaN flag. The input is slightly insane. """ dest_path = "data/ROME-FIELD-{:02d}_quad{:d}_photometry.hdf5".format( field, quadrant) srchdf = h5py.File(rd.getAbsPath(dest_path)) _, arr = next(iter(srchdf.items())) try: photpoints = arr[quadrant_id-1] except IndexError: # test instrumentation: test set construction currently is # such that index 1 in the test hdf5 is 70001 in the original. # Yeah, it's a pain, but so is the upstream format base.ui.notifyWarning("Rome: test data kicking in") photpoints = arr[quadrant_id-70002] photpoints = numpy.array(photpoints) photpoints[photpoints==0] = numpy.nan photpoints[photpoints==-9999.99] = numpy.nan return photpoints @functools.cache def get_photpoint_char(field): with base.getTableConn() as conn: return list(conn.queryToDicts( "select filename, hjd, filter" " from \schema.images" " where field=%(field)s" " order by image_index", locals())) def get_photpoints_with_bands(field, quadrant, quadrant_id): """yields a sequence of (bandname, hdf5row, src-image) tuples. Completely broken hdf5rows are already swallowed here. It's a shame this function is even necessary. Bandname is one of i, r, or g. """ photpoints = rd.get_photpoints(field, quadrant, quadrant_id) images = get_photpoint_char(field) for rec, imgrow in zip(photpoints, images, strict=True): # Since we replace 0 with NaN, ok data points now have NaN # in their qc_flag. Sigh. if numpy.isnan(rec[-1]): continue # chuck out all records without usable photometry, too. if numpy.all(numpy.isnan(rec[1:9])): continue assert abs(rec[0]-imgrow["hjd"])<1e-7 yield imgrow["filter"], rec, imgrow["filename"] def parse_rome_id(rome_id): """returns a tuple of field, quadrant, quadrant id from one of our internal rome_ids. """ return rome_id//10000000, (rome_id//1000000)%10, (rome_id%1000000) rd.get_photpoints = get_photpoints rd.get_photpoints_with_bands = get_photpoints_with_bands rd.parse_rome_id = parse_rome_id qc_flag is a bitmap, where 0 means a good image. In the light curves we serve, only points with qc_flag=0 are retained, and you should only use other data if you really know what you are doing. In case you get ROME data from other sources, here is what the flags mean: ==== ================================================================= 1 No valid photometric measurement possible by stage 6 2 Image photometry displayed excessive scatter (mostly bad seeing) 4 Data point failed phot scale factor/exposure time metric 8 Low quality image resampling 16 Difference image exhibited high residuals ==== ================================================================= Elements to arrange for parsing from the 5-ext FITSes. Pass hdu to select the extension, as well as accref, table, fsize, mime, and path for the products rowfilter. All the latter are python expressions. And there's rome_id, a python expression to compute a value ending up in a field rome_id. mat = re.search(r"([0-9]+)_field", sourceToken) return { "field": int(mat.group(1))} @rome_id = \rome_id yield row \accref \table \fsize \mime \path \preview \preview_mime "filter" { "ip": "SDSS i'", "rp": "SDSS r'", "gp": "SDSS g'",} Time series observed with ROME/REA in a custom scheme.//ssap#plainlocation//ssap#simpleCoveragerome_id \banddefs make-view 0.0002 @ra @dec sdlssa_pubDID//ssap#view//obscore#publishSSAPMIXC \banddefs ROME/REA time series in the \filterHuman band. //timeseries#phot-0 [ # this is headings for the HDF5 fields. Sigh! "hjd", "instrumental_mag", "instrumental_mag_err", "calibrated_mag", "calibrated_mag_err", "corrected_mag", "corrected_mag_err", "normalized_mag", "normalized_mag_err", "phot_scale_factor", "phot_scale_factor_err", "stamp_index", "sub_image_sky_bkgd", "sub_image_sky_bkgd_err", "residual_x", "residual_y", "qc_flag"] { "SDSS g'": "g", "SDSS r'": "r", "SDSS i'": "i",} d = self.sourceToken for filter, rec, filename in rd.get_photpoints_with_bands( d["field"], d["quadrant"], d["quadrant_id"]): row = dict(zip(column_labels, rec)) row["image_link"] = ("https://archive-api.lco.global" "/frames/?limit=1&proposal_id=KEY2017AB-004&basename_exact=" +urllib.parse.quote(filename[:-5])) yield (table_code[filter], row) We're building three tables in parallel; this is the common mapping stuff. obs_time: hjd, phot: normalized_mag, phot_err: normalized_mag_err ROME/REA Datalink Service data "\rdId#time_series" "#timeseries" "\rdId#build_ts" from gavo.protocols import sdm return sdm.formatSDMData(descriptor.data, "application/x-votable+xml;serialization=TABLEDATA;version=1.6") field = descriptor.ssaRow["field"] quadrant = descriptor.ssaRow["quadrant"] yield descriptor.makeLinkFromFile( f"data/ROME-FIELD-{field:02d}_field_crossmatch.fits", description="FITS tables containing metadata for the observations" " of this field.") yield descriptor.makeLinkFromFile( f"data/ROME-FIELD-{field:02d}_quad{quadrant}_photometry.hdf5", contentType="application/x-hdf", description="An HDF5 file with photometry for this time series'" " quadrant of the field (you have to consult the documentation" " to understand this).") yield descriptor.makeLink( "https://github.com/rachel3834/romerea_toolkit", contentType="text/html", description="Tools for handling the ROME/REA survey data.") yield descriptor.makeLink( "https://ui.adsabs.harvard.edu/link_gateway/2024PASP..136f4501S/PUB_PDF", contentType="application/pdf", description="The data release paper describing the upstream data" " products linked to here.") ROME/REA preview maker rome_id try: photpoints = rd.get_photpoints_with_bands( *rd.parse_rome_id(int(inputTable.args["rome_id"]))) except Exception as exc: raise svcs.UnknownURI(f"No such preview ({exc})") series = {"SDSS r'": [], "SDSS i'": [], "SDSS g'": []} for filter, row, _ in photpoints: series[filter].append((row[0], row[7])) with processing.matplotlibLock(): fig = figure.Figure(figsize=(4,2)) ax = fig.add_axes([0,0,1,1], frameon=False) for band, color in [ ("SDSS i'", "red"), ("SDSS r'", "green"), ("SDSS g'", "blue")]: points = series[band] ax.plot( [p[0] for p in points], [p[1] for p in points], "o", color=color) rendered = io.BytesIO() fig.savefig(rendered, format="png", dpi=50) return ("image/png", rendered.getvalue()) \schema Web ROME/REA DR1 Web accref, hjd_min, hjd_max, mag_i_min, mag_i_max, np_i, upstream_name \schema SSAP full auto ssa/ssap.xml row = self.getFirstVOTableRow() self.assertAlmostEqual( row["location_ra"], 267.5934298564758) self.assertEqual(row["rome_id"], 11070003) self.assertTrue(row["accref"].endswith("/getproduct/rome/ts/11070003")) # TODO: Test for datalink presence sdl/dlmeta by_sem = self.datalinkBySemantics() self.assertTrue(by_sem["#this"][0]["access_url"].endswith( "/getproduct/rome/ts/11070003")) self.assertEqual( by_sem["#this"][0]["content_qualifier"], "#timeseries") self.assertEqual( by_sem["#documentation"][0]["access_url"], "https://github.com/rachel3834/romerea_toolkit") self.assertEqual( by_sem["#progenitor"][0]["description"], "FITS tables containing metadata for the observations of this field.") self.assertEqual( by_sem["#progenitor"][1]["access_url"][-10:], "_photometry.hdf5"[-10:]) sdl/dlget 2458586.175836738", # first HJD "17.482026942113468", #first phot "basename_exact=coj1m011-fa12-20190412-0164-e91" #image name ) ]]> /getproduct/rome/ts/11070003?preview=true self.assertHasStrings("PNG", "Matplotlib v") /rome/q/web/form self.assertHasStrings( "15.705", #min mag or 01_287895 "235", # N(i) "/q/sdl/dlmeta?ID=ivo://org.gavo.dc/~%3Frome/ts/11070002")