Following are my meeting notes from the Real Estate Standard Organization’s 2016 Fall Conference. It isn’t meant to be comprehensive, but covers what happened at key sessions for those who couldn’t attend. Some of the presentations should be available at some point on the RESO website.
Jeremy Crawford, Executive Director for RESO kicked off the RESO meetings by introducing new attendees to RESO, which has the vison of “A streamlined real estate technology industry”. For those not familiar with RESO, its mission is “To create and promote the adoption of standards that drive efficiency throughout the real estate industry.” RESO is a non-profit, member organization with categories for NAR, tech vendors, brokers AORs/MLSs, and non-profit entities. Committees are driven by volunteers and ratified standards are free and open source. The standards process starts with the Research & Development group, which acts as a funnel for new ideas for the standards, then through other groups like Data Dictionary, Transport, Internet Tracking, Property Unique ID, Payloads, then finally through the Technical Committee to the Board of Directors for final standards adoption. There are also certification and marketing committees.
Workgroup meetings on the first day included:
- Payloads Workgroup. The agenda is to review the IDX payload the group has developed, and discuss certification (e.g. an IDX “Endorsement” to the standard certifications).
- Property Unique ID Workgroup. This group is building from efforts form the US government, public records, and other data standards organizations to build the “definitive” standard for the Universal Property Identifier to be used by the real estate industry. The agenda included a history of the effort, an overview of the proposed PUID structure, industry concerns, a solution approach, and project timelines.
- Data Dictionary Workgroup. This group is creating a standard for the fields and lookups (enumerations) in the MLS. It is a common center for all expressions of fields and enumerations in the RETS standard. The workgroup purpose is to maintain and develop the Data Dictionary. Agenda included a process review, a discussion of the December enumerations meeting, the commercial property subtype, ownership / structure & PropertySubType, and “[Type]” Enumerations, Saved Searches, Contacts, Auto-Email relationships, and version 1.6 strategic objectives.
- Internet Tracking Workgroup. The charter of this group is to create analytics that show value for tracking activity around the MLS listing – ideally creating a unified view of activity across all the systems where the listings are displayed. The agenda included a field summary, transporting the fields, tracking and recording methods, and next steps.
- Research and Development Workgroup. This workgroup solicits and reviews submitted business cases from the real estate community and identifying how RESO can contribute to the benefit of that business process. The agenda included a review of current initiatives, a presentation about SSO and account linking, lockbox activity, standardizing showing data, metadata change history, industry best practices (CMLS) and MLS Business Rule Expression.
- Transport (Web API) Workgroup. The agenda included discussing RESO Web API Update, Updated goals, use cases, oData Support, open discussion and next steps.
Other sessions include “Taking your brokerage to the next level using RESO standards”, a “Show N Tell” contest with 20 entries, and more.
Tragically, various workgroups met at the same time, so this summary will not be comprehensive. Here are some of the details of those workgroup sessions:
Property Unique ID Workgroup Session
The workgroup chair is Mark Bessett from Arizona Regional MLS. The goal of this workgroup is to create ubiquitous identifiers for properties. The standard must be standards based, non-proprietary and straightforward to implement. It must be portable – the universal key to an ecosystem of data. There has been similar work by the U.S. Department of Transportation and the CFPB, and this group has found many ways to represent uniqueness. There are many use cases, but one of the most important is linking records like tax history, listings, mortgage history, ownership history, and insurance claim history. It would also help listing aggregators identify and perhaps eliminate duplicate listings.
The proposed ID is based on public records.
- It’s based on an ISO convention model (hierarchical)
- ISO 3166 specifies country codes
- Specifies that country identify sub-regions
- US uses FIPS conventions for unique county
- Most or all counties have unique numbering for parcels
There’s an equivalent standard for Canada.
|Parcel Type Identifier||County defined Parcel Number||Other Differentiator? (reused?, etc.)|
The workgroup needs help, especially to figure out those differentiators (disambiguators). Also, to evaluate current concerns:
- Is the county uniqueness assumption valid?
- Do some areas allow APN re-use?
- Do delimiters present challenges?
- Are APNs governed (resilient to change)?
The solution approach is to:
- Identify proposal issues (listed above) 12/2016
- Verify and validate (Zillow, CoreLogic, CRES Data granting research license to RESO) 4/2017
- Finalize ID components 6/2017
- Launch proof of concept for testing 7/2017
- Public PUID definition standard 9/2017
- Launch RESO Open source Repository (returning PUID only) 11/2017
The group discussed issues with parcel splits, changing APN (based on land use) in California, and more. Yes, if the property changes in a substantial way, the PUID is going to change. In the case of apartments buildings with only one APN, there will need to a “unit” disambiguator. In commercial there’s a lot of “assembly”, when 15-20 blocks of residential property may merge to a single commercial APN – that will need to be accounted for – starting the disambiguator with a zoning key may be desirable. There may need to be a way to track those APNs and PUIDs that got “recycled”.
Matt says: There needs to be a standardized way of expressing lineage if we want to use this as a tool to track what has happened to a property over time, even as PUIDs change (due to parcel splits, assembly, etc.). This could potentially be accomplished using Blockchain technology.
The PUID Home is at http://members.reso.org/display/PUID
Data Dictionary Workgroup Session
Rob Larson, the workgroup chair, started with a process review:
- Suggestions come through the Wiki from reso.org. We’re no longer using a spreadsheet.
- Posted to Confluence for discussion (members.reso.org).
- Managed in confluence tasks by administrators
- Changes will be discussed with those already using the field (via the certification team)
- There is a focus on stability around changes, and a process to support that.
The group will be meeting in San Dimas – December 6-7. The group will be working on the gold enumeration worksheet. There is a google doc where members can contribute their enumerations, whether they are going to make it to San Dimas or not.
There is a posting on Confluence regarding Commercial PropertySubType
- Long and short lists
- Sub Groups to the Sub Type
- Propose an extendable “medium list” (staying away from business types – no “donut shop” – that’s “Business Type”).
- Add to PropertySubType
- Individuals are okay to create a sub sub type.
Examples are: agriculture, business, hotel/motel, industrial, mixed us, multi family, office, retail, unimproved land, and warehouse.
Some discussion items were brought up regarding this enumeration:
- Would industrial include manufacturing, or would that be a separate list item?
- What about commercial improved / commercial unimproved? This could be extended on an individual MLS level – or if there are enough MLSs that need this, it could be added to the list.
- There may need to be more vetting around farm vs. agriculture.
- How do we deal with a property subtype in RETS (e.g. multi-family or farm) are full-on property types in an MLS? Rob Larson discussed how a property might be cross-listing between Commercial subtype Multi-Family and the property type Residential Income as an example.
- How about “medical”? The list could be extended – start chiming in on the list on Confluence!
The current enumeration passed voting – note that this doesn’t mean that it can’t be extended further via further work in Confluence.
The conversation next moved to Ownership and Structure, and the idea of a gradual replacement of the property sub-type.
- Describe the type of ownership (e.g. fee simple (freehold) or lease (leasehold)) – NOT ownership structures (individual, partnership, corporation, trust) and NOT forms of transfer/financing (deed, right to use, in acquisition).
- What about types of common interest (coop, condo, etc.)? RESO’s property subtype includes condominium, stock cooperative, timeshare. Other common interests include planned development, community apartment, or none. Canada has strata and cooperative.
- Ownership option 1: keep existing LandLeaseYN and add new field CommonInterest (condominium, planned development, community apartment, stock cooperative, timeshare, none) where LandLeaseYN=No means fee simple / freehold.
- Ownership option 2: Deprecate LandLeaseYN and replace with Ownership Type (land fee simple, condominium, planned development, community apartment, land leas, stock cooperative, timeshare).
- How about adding tenants in common (TIC)?
- What direction should the data dictionary go? Option one was selected.
- DO describe the types of structures (SFR detached, apartment building, townhouse/rowhomes) but NOT styles of architecture.
- Do we keep PropertySubType around, or add Structure type?
- Enumerations: Single Family Residence, Manufactured Home, Townhouse, Apartment (condo?, unit?), Duplex, Triplex, Quadraplex, Boat Slip, Deeded Parking, Cabin, Farm, Ranch, (+ our commercial list).
- “SFR is too generic – all my agents will just pick that one.”
- SFR is NOT a structure – that’s just how it’s used. Try “attached” or “detached”.
- There are really two things reflected in this list at present – what it is and how it was built. That may be a separate action item, if desired by the group.
- This needs to be vetted out more.
How should we enumerate repeating elements?
- Pretty Easy (one size fits all)
- Features can be different for each type
- Should RESO tackle the “low hanging fruit?” (e.g. bedroom, bathroom, kitchen, other?)
- Bedroom & Bathroom
- One list, or different lists for rooms 1-5 (and master?)?
- Other Fields
- Basement, Bonus Room, Den, Exercise Room, Family Room, Game Room, Great Room, Gym, Laundry, Library, Living Room, Loft, Media Room, Office, Sauna, Utility Room, Workshop
The group voted to move forward with an exploration of the second option.
Research & Development Workgroup Session
Greg Moore, the workgroup chair, introduced the group and how it works – promoting use of the online discussion system on reso.org, and encouraging a rich conversation that drives standards development. We talked about the following subjects:
- SSO Account Linking. Paul Hethmon from Clareity Security presented a proposal for “User Federation”. More on this subject later in this post.
- Lockbox activity payload. Summary: Work with RESO member lockbox vendors, MLSs and brokerages to define a standardized data feed (payload) for lockbox data. Greg has reached out to some vendors already. The value is having current standardized lockbox activity data available to brokerage and MLS systems would facilitate better statistics. RESO member lockbox system providers collaborate with the broker community and MLSs to define a standard (data dictionary) field set and definition for a lockbox data feed. This involves lockbox system providers supplying data and brokers and MLSs who consume the data.
- Standardize showing data. This is similar to the lockbox effort – in this case, facilitating updates to broker calendaring systems and facilitating better statistics.
- Metadata change history. The idea is to create a payload so people can see what RETS data structures have changed over time or are going to change.
- We have created some Best Practices documents with CMLS
- Basic RETS Usage. At the heart of it, for providers: Ensuring data is current and accessible via compliant RETS server. Ensuring brokers can access their complete content. Ensuring all data consumers of RETS are aware of certified resources available. Each resources should have a record modification date. There should be clear documentation of available resources. Deleted listings, media and other content is clearly documented. Provide 4 weeks written notice of significant changes, providing documentation. Providing access to a test platform at least 2 weeks before release. Data consumers need to know the RESO certifications the data provider has achieved, to ensure data pull are efficient, to comply with data access and display rules, and pull the payload specific to the data use case. The full best practice document will be published by CMLS.
- Discuss developing an image (media standards) best practice…
- Review other business cases at the reso.org confluence R&D home page.
- Business Rules syntax. I described the basic syntax as further described in the full documentation available on the reso.org collaboration system. Susie Cass from RPR described her experience using the syntax to successfully document MLS rules.
- The following slide illustrates WHY it’s best to use structured English and REBR rather than programming code to exchange rules. The proprietary code is unintelligible to anyone but the originating vendor, and cannot be validated by the business folks!
Transport Workgroup Session
Greg Lemon took notes from this session that I could not attend due to the concurrent scheduling of workgroups. This is my summary of those notes and any mistakes are mine.
It is the objective of this workgroup to have an Update Standard (draft) completed in Q1 2017 – that standard will allow for API-based listing maintenance. There were still questions remaining about what will be included in this initial release – whether it will focus only on new records or on modifications to existing records; also whether it will include partial, full, or a minimal field set for update transactions (at first). The consensus was to start out with updating media as a starting point to learn lessons that would be applied to other resources. Also, it was determined that the location of the Business Rules (BR) logic (at this time) should be behind the API and reside on the server. [Matt’s note: the BR payload should soon be sent to Transport, so they can figure out the appropriate way to expose this payload.]
The group discussed OData support, and the workgroup will look through the OData 4 standard to see which sections (e.g. OData version 4 Section 11.4) will be used by each of the use cases. Hopefully RESO will not need to deviate from the standard, or not very much.
Payloads Workgroup Session
What I heard from several people that attended this session is that there was a lot of tension around the idea that there would be a minimum set of fields that MLSs might be required to include in order to get an IDX Payload “Endorsement”. At one point during the meeting MLSs were assured they may not be required to include specific content in the IDX payload fields, but if so it’s unclear what the value of this payload would be for brokers. It sounds like this is going to be an ongoing discussion.
Jeremy Crawford kicked off the day. RESO has undergone great growth in the past year. There are now over 570 RESO member organizations contributing to standards – a 30% increase year to date. There’s a 25% overall increase in workgroup participation. Over 1.2 million MLS subscribers are represented by 630+ MLSs that are data dictionary certified. Over 375,000 MLS subscribers are represented on MLSs that are Web API Server certified. The conference this year had 32% more registrants than last year.
Art Carter talked about strategic positioning for RESO and its role in the real estate industry. One of the main goals for the board of directors has been to improve the velocity with which standards are developed, and this goal has been achieved. Art talked about how this organization is a listening organization, and he used the Payloads workgroup as an example of this – he recognized how much brokers from that session wanted to see a standardized IDX payload. Art talked about the composition of the board of directors – he encouraged people to visit reso.org to learn more about the organization. The nomination period for the board of directors will end November 10th. Finally, Art talked about the aggressive strategic plan and how hard the RESO staff (like Jeremy Crawford, Greg Lemon, and the staff from IMI) have worked to meet goals.
Shaun York from Homes.com talked about how they perform listing data aggregation. In terms of Inventory, Homes.com manages over 5.8 Million active listings from 834 sources: 643 MLSs, 70 brokers, and 122 others. They use RETS (646 sources), FTP (push and pull – 129 sources)), HTTP, and Web Services to get these listings. Most are updated hourly, most of the rest at least daily, and a few (40 sources) less than once per day. Homes.com has a “Discovery” pipeline that goes through several stages: image processing, geospatial targeting, property record matching, school and neighborhood tagging, data deduplication, profile construction, and publication. Altogether that’s 733 operations per second, just to process listings! Each listing takes 5 seconds to go through the process, with most of that time being image retrieval. Over 65 organizations receive Homes.com performance metrics, in formats including 3rd party APIs, FTP flat files, and even PDFs attached via email. Measures are both real time and aggregated over a period of time (day, week, or month). Shaun shared a list of the fields they typically share, and expressed the need for the standardization currently being worked on by RESO. Homes.com would love to reduce the use of non-RESO transports, and leverage the new RESO Web API further.
There were a number of other sessions (including the “Show N Tell” on day one, a panel I spoke on, and other day two sessions) that were informative and fun but weren’t really focused on RESO’s mission of creating and driving adoption of data standards, so I’m not going to cover them in depth in this (already) very long post. That said, it certainly was interesting to see how Zillow wants to compete with what Upstream is doing –, and how many proprietary APIs they have (agent reviews, leads, AVM, public records, traffic/tracking) that they might want to coordinate on with other vendors via RESO to create standards. On day three there was also a presentation by Ethan Bailey about CoreLogic Trestle which was, in some ways, comparable with the Zillow and Upstream presentations but also adding a data marketplace and contract workflow. But again, detailed descriptions of those presentations are really out of scope for an article about data standards development – perhaps in another article!
Rob Larson from CRMLS presented “Data Dictionary: Then, Now, Beyond”. In 2011, 23 MLSs met to look at RETS Standard Names and their own contributions. By 2012 they released version 1.1 of the Data Dictionary that had 731 fields, 65 enumerations, covering the following resources: property, member, office, media, history, and contacts. By August 2013, they had 801 fields and two new resources including open house. In version 2.3 released in July 2014 the workgroup added commercial classes, defined “core fields”, added 42 fields, and 93 modifications. In August 2015, workgroup released version 1.4, which had five certification levels, a “teams” resource and over a hundred modifications from the previous version. In June 2016, the workgroup released 1.5 with 1058 fields and 1434 enumerations. Work was done in a Wiki rather than a spreadsheet; they added the OUID resource, minimum IDX fields, fields about power production and a “Silver Enumerations” level.
In Spring of 2017 the group will release version 1.6. Work will be done to evolve ownership/structure versus property sub-type, adding accessibility and showing data fields, “Gold Enumerations”, and resources for saved searches, carts, and portal preferences, as well as Internet Tracking Fields (from that work group). Beyond that, in version 1.7, the group will finish enumerating picklist fields, add a system rules resource, and property unique IDs. Additional targets (for 2017/2018) include transaction management, tax, statistics, evolving tech (VR), teams (enhanced), CRM, back office, Internet of Things, and DOE / Green Data Sources.
“Toward RESO standard real estate developer platform for MLS and Brokerages” was presented by Dr. Umesh Harigopal and Matt Kumar. Matt says: If you aren’t “geeky”, skip to the next major section of this blog, “Day Three”! They believe the opportunities to accelerate analytical innovation are in three areas:
- Big Data & Integration
- Longitudinal Property Record
- Common temporal model for real estate properties
- Single standard canonical model across data sets
- Assessment, Recorder, Neighborhood, etc.
- Retrieval & Delivery
- Standard Real Estate Ontology
- Matt says: Ontology is a formal naming and definition of the types, properties, and interrelationships of the entities that really or fundamentally exist for a particular domain of discourse. I know, that’s still pretty dense stuff.
- Represent RE knowledge for effective querying.
- Enable natural language processing (NLP) frameworks extract ontological structures
- Analytics & Insights
- Reusable Analytical Models
- Appraisal models, improving consistency
- Real Estate Cognitive Computing framework
Delivering a standards-based cognitive computing platform provides a “cognition fabric” providing insights – potentially predictive – from unstructured and structured data streams. This would involve an application layer on top of standard frameworks & platform (realtime predictive models, cognition fabric, longitudinal property record, and canonical data model). That would sit on top of image, video, text and transactional data.
Focusing on the predictive angle, they are trying to generate the following insights:
- Recommend when to sell a home
- Recommend who to work with to buy/sell my home
- Owner’s propensity to sell
- Best home to buy
- Predict a home price
- Recommend best investments
- Semantically inconsistent data
- Partially specified data
- Integration of disparate information domains
- Natural language with incomplete sentences.
This group has a “solution approach” to evolve metadata to ontology, taking all the information and putting it through natural language processing and data integration, leveraging machine learning models and RE Ontology, along with query and analytics engines and a derived knowledge graph.
The example they provided involved pricing predictions. Neat stuff!
Jeremy Crawford kicked off the day by urging MLSs to switch their data feeds over to use the Data Dictionary, even knowing how difficult it may be to do so for all involved.
Glen Phillips, CEO of Lake Homes described “The Good, The Bad, and the Ugly: Lesson from Joining 50 MLSs in 80 Months”. Some background: How could a small brokerage compete with national ones (using a “Money Ball” movie analogy) – by creating a highly virtual company, with one foot in the cloud and one in bricks in mortar – just like Uber, Amazon, Airbnb, Instacart, and Shipt. This brokerage focused on Lakes – which stretch between the MLSs, and they needed to create a single website for consumers, on which they could find all lake properties, regardless of MLS. They’re in 50 MLSs in 10 states, focusing on 41,000+ listings – specializing on specific lake inventory. Solds and pending listings are only available for display from a limited number of MLSs and not all offered a data dictionary feed. The good: There are service-minded MLSs and dictionary certification appears to have buy-in, and almost all MLSs have good data. The bad – sold data in feeds is NOT commonly available and specialized data field usage is very inconsistent (e.g. agents not marking a listing as lake view, lake access, etc.). The ugly: some MLSs are unresponsive to members and some rural-area MLSs commonly are data isolationists. A few MLSs operate to fund staff, parties, and trips. Two MLSs they deal with issue “policy” fines in direct conflict with laws and regulations. Overall, there’s more good than bad/ugly. With time, they are typically treated professionally. Consumers want more data – this is an opportunity! Matt says: I think this presentation really plays into both MLS rules and how NAR monitors MLS compliance – as well as informing our ongoing discussions of resources and payloads in the process of standards creation at RESO.
Dave Duran from Yo Data talked about creating a Schema.org extension for real estate data based on RESO standards. According to Dave, Google is using this type of structured data to create search results, displaying key data in their interface. This is illustrated in how they display recipe information, travel information, and more – above the organic search results. Zillow, Trulia, and Redfin are taking advantage of this. He suggests extending schema.org for real estate data, specifically open house events, real estate offers (who is selling something), and identity (in terms of the author/ originating publisher, e.g. the listing broker). Dave wants to see if he can get RESO to participate in this effort.
Scott Petronus from Onboard Informatics and chair of the Transport Workgroup, along with Demian Quinonez from CRMLS, Ashish Antal from MLSListings, and John Kennedy from dynaConnections presented “Implementing the RESO Web API: Challenges and Lessons Learned”. A number of challenges came up including authentication and security – whether to use Oauth2 or OpenID Connect. Matt says: I believe how that’s going to work out in the final-final spec down the line is still a matter of discussion!). Certification of OpenID Connect security was challenging for one panelist because there was no one at RESO to talk to about how to meet those requirements. Other challenges included getting metadata for the data dictionary right, and implementing specific search features like date/time. Another good question came up – “How do you charge for this service? On the basis of “hits”? How do you monitor usage?” That’s going to be an ongoing discussion. While there were a few bumps in the road for certification, it all worked out for all of the panelists. One of the panelists suggested there be a reference implementation set up – not the first time this suggestion has come up during the standards development process. Another was very concerned about how top-heavy OpenID Connect is, and the lack of industry support for it (again, this is an ongoing discussion).
Paul Hethmon from Clareity Security proposed a process for user federation using both SAML and OpenID Connect protocols by using a relying party proxy. In a nutshell, this is a process for allowing users to move from one group of SSO-connected system to another, where for example an MLS may have one SSO “identity provider” and a broker has another. When a user moves from one to the other, the second would communicate back to the first, and the accounts would be linked. In that way, for example, the broker intranet would know on an ongoing basis that user “Paul” coming from the MLS is the same as user “PaulHethmon” on the broker system when that user moves between systems. Paul has provided a formal proposal for this, which is on the RESO site, in the R&D area. One person in the audience asked, “Why not use OpenID to do this?” the answer for which is that we need a platform agnostic solution because people use different SSO protocols – which is exactly why this is a discussion for the standards group.
Matt McGuire from CoreLogic presented “The Final Planned Version of the RETS Specification”. All proposed changes will be made by January 1, 2017. There will be a workgroup review, and final submission to the board of directors by mid-March 2017, if not earlier. Testing requirements will need to be nailed down early, so testing and certification goes smoothly. Highlights of 1.9 include:
- Additional support for data dictionary and WebAPI transition
- Update standards references to current versions – improved support for 3rd party libraries and security standards
- A potentially breaking change in RCP 108 with change to headers to use ‘X-‘ form (e.g. X-RETS-Request-ID)
- Further corrections to the standard including Update Transaction.
Note that there was no RETS 1.x meeting during this conference. RESO is no longer just “RETS” – it’s so much more now.
That’s all folks! I hope you find my notes useful.
Share this post: