Categories
Blog Design Reviews

Website Redesign Report: ABA Free Legal Answers

This report is from our student group in the Spring 2023 class 806y, Justice By Design.

American Bar Association’s Free Legal Answers: User Experiences and Recommendations 

by Sonya Googins, Justin Iannacone, Kelsea Jeon, Shannon Lee, Ana Ribadeneira, Kevin Wang, June 9, 2023

Introduction

Through interviews and research on how to improve access to free legal services, our team has come up with recommendations for ABA’s Free Legal Answers (FLA). Given that the ABA’s main objective is to improve outreach efforts for FLA, our team focused on enhancing the user’s experience with the site by boosting its accessibility, transparency, and trust. Our memo follows with the assumption that the first step to creating a strong outreach strategy is to improve the individual user experience with the existing site. That way, when prospective users eventually learn of the resource and use it, they are confident that FLA is a reliable and trustworthy source of help. 

Overview: User Experience with FLA 

This memo is organized through the lens of a user’s experience with FLA. We categorize this paper into four main steps:

(1) Arriving on the Home Page;

(2) Determining One’s Qualifications and Suitability;

(3) Submitting One’s Question(s);

(4) Receiving the Response.

The final section contains recommendations for data-driven next steps. 

Touchpoint: Arriving on the Home Page

This section of the memo focuses on the user’s experience when they arrive on the home page. First impressions are key; you never get a second chance to make a first impression. As website designs grow more sleek and modern, we think it is critical to greet users with a landing page that is visually welcoming, functionally accessible, and aesthetically professional. With these goals in mind, we propose changes to six aspects of the landing page.

Updated Home Page Cover Image and Text

Before: Current webpage visual

After: proposed new image

The existing FLA homepage displays a photo of raised hands. We recommend updating this background image and the bold text on the homepage. Regarding the background image, users during our interviews reported that the main page image of the raised hands could use updating; and we agree. We created a mock-up of a new homepage cover photo that emphasizes the strength of the previous cover photo–an emphasis on diversity and a message of being welcoming to all, regardless of background.

Hence, to maintain the welcoming feel of the site, while updating the image aesthetically, we substituted the image of the hands with a more modern image of a diverse group of people looking off in different directions for help.  During our interviews with stakeholders and potential users, interviewees also noted that the bold text, “Can’t Afford a Lawyer?” seemed to suggest that the site would offer users an attorney to represent them. To avoid confusion about what FLA offers and to more clearly state the site’s purpose, we recommend replacing the text with: “Get legal answers.”

Updating the Navigation Tabs to be User-Centered

Before: The Existing Navigation Tabs

After: New Possible Navigation Tabs


Another area on the main home page that we recommend updating is the tab bar. We suggest visually updating the tabs at the top of the home page, as well as modifying some of the pages, so that users are offered more information as they arrive. The “Legal FAQs,” “User Reviews,” and “About Our Volunteers” pages, all of which are new changes that we will discuss below, will be helpful for new users who are unsure about what the site offers and whether the site is suitable for their needs. Making sure such information is available immediately will also ideally increase user retention rates. We have grayed out the “Attorney Registration” tab to signal to users that the tab is not for their purposes. This slight modification will serve to make the site less attorney- and volunteer-centric and more user-based. 

Language Translation options

In an effort to better retain users of color, we recommend implementing page translation options. Many individuals who need pro bono legal help are first-generation immigrants, many of whom do not speak or read English. The languages we suggest including on the website are Spanish, Mandarin Chinese, Korean, and Vietnamese, which are spoken by large immigrant groups in the United States with high percentages of non-English speakers.

Currently, there are several online programs and browser extensions that can translate pages from the user’s end, such as Google Translate. But because translation this way requires the user themself to download translation programs, we believe it would be more accessible for those who are less technologically savvy for translation options to already exist on the page like in the example above. Making sure that translation is just a visible click away will hopefully maximize retention rates for non-English speakers. Indeed, we realize that translation options will not be useful if the legal answers that users receive are still in English.

Thus, we recommend that the ABA make a concerted effort to try to attract volunteers that can speak–and write in–languages besides English.

About the Lawyer Volunteers

Another homepage tab we suggest creating is a tab for our users to learn more about the lawyers who volunteer for FLA. One of the strengths of FLA–as opposed to other online resources of this nature–is that users can receive help from a real lawyer with real expertise.  However, this feature is not being marketed as clearly or broadly on the existing site. Once users realize that they are receiving human-centered, individualized assistance, they might be more included to trust the FLA services as well.We suggest curating a few biographies and personas about the attorneys to allow users to get to know more generally who is on the other end of the FLA service. The following images are examples of what a new “About Our Volunteers” tab could look like.

These new features could be integrated into the “Volunteer Recognition” tab that is already on the site. On these pages, FLA could include direct quotes and testimonials from lawyers who volunteer for FLA. In these testimonials, volunteer attorneys can answer questions that users we interviewed had, such as: Who are the lawyers volunteering with FLA? Why are they volunteering? Why do they think FLA is a good resource for me to use? Personal information about the volunteer attorneys can be anonymized to protect their privacy. 

User Testimonials

Our next recommendation is to include a “user testimonials” section in a separate tab on the website. One of the most common pieces of feedback we received from stakeholders and potential users was that while the website seemed to offer an excellent service, users wanted to know other users’ experiences with the website. For instance, a potential FLA user stated that she wished she could read Google reviews of others’ experiences with Free Legal Answers.

Because first-time users are likely unfamiliar with the website, user testimonials can be one way to gain users’ trust and reassure them that this is a legitimate service that helps real people. This is especially important given that FLA is a free service, and users have often expressed skepticism about such services, thinking that it is “too good to be true.” And of course, when posting these testimonials, the information on the website can be anonymized to protect individuals’ privacy. An example of one such user testimonial is included below. 

State-Specific Landing Pages

Before: current state landing page

After: proposed new state landing page


Finally, once the user gets past the home page and selects the state in which they reside, the current FLA page takes them to a state-specific landing page, which is the next page in the user’s journey. We suggest updating this landing page in three ways.

First, the “Get Started” button should be centered to better capture users’ attention as the way forward if they wish to proceed. Second, we suggest moving the “Other Places to Get Help” link into its own button on the bottom left. On the current site, the link is difficult to see—many users in our interviews did not see the link when exploring the page. Lastly, on the bottom right, we recommend incorporating a link to a new proposed feature, “Legal FAQs.” We describe the details of this new feature later in this report.

In response to potential concerns about search engine optimization, this FAQs page–or a more general version of it that is not specific to states, but can be applied to legal help and courts more generally, can also be put on the home page. 

Touchpoint: Determining Qualifications and Suitability

Once a user has decided that the website is trustworthy and suitable enough to go forward, the next stage is to help the user better understand how to use FLA. After a user clicks the “Get Started” button on the state-specific landing page, we propose that the page redirect the user to a new page. Currently, when one decides to “get started,” they are taken directly to a user agreement. But before that, we suggest the resource direct the user to a general overview section with information on how FLA works, how to submit a question, and how long the process takes. For instance, the current website does not indicate how long it takes to receive a response from a lawyer, making it difficult for users to have realistic expectations for how long they have to wait before receiving an answer. Providing greater transparency as to what happens to a user’s question once it is submitted will help build trust with users and allow them to better allocate their time if they know they may have to wait a set number of days before receiving an answer.

Now, after getting a broad understanding of how FLA works, the user decides to proceed with the service. They are then taken to a user agreement page to see if they qualify to use the service. The primary issue with the previous user agreement page was that users had difficulty determining whether or not they qualified to use FLA. Specifically, the biggest sources of confusion included determining whether or not someone was “low-income” and what it meant to have “low” balances in one’s financial accounts. The second issue with the previous user agreement page was that the agreement itself was too text-heavy, making it difficult for users to identify the important requirements in the agreement. 

In light of these concerns, we propose two main changes. The first change addresses specific areas that users pointed out during our interviews that caused confusion when determining whether one qualified, and the second change addresses the general readability and format of the user agreement. 

The above mock-up is our proposed version of the new user agreement. The areas circled in red represent new additions to the existing text of the qualifications. Below are explanations for these new additions. 

First, the phrase “cannot afford a lawyer” clarifies what it means to be low-income in a way that is relevant for using FLA, because it emphasizes that an individual is considered to be low-income when they cannot afford an attorney. Although some stakeholders have expressed concern that the “cannot afford a lawyer” phrase is also ambiguous, we believe it is clear enough for users to determine whether the service is right for them. Ultimately, the ABA is balancing the desire to help those with pressing civil legal needs with concerns from attorneys who would otherwise serve clients who are able to pay for legal services.

FLA can balance both interests by asking people whether they are low-income and unable to afford an attorney. If someone cannot afford an attorney, they would not be able to hire an attorney in the first place and would thus not be unfairly taking advantage of FLA. Another reason why the “cannot afford a lawyer” qualification is preferred is because it takes into account the cost and availability of lawyers in an individual’s state. Legal markets can look very different across the country, and by allowing users to determine whether they can afford a lawyer, it allows those in the best position–users themselves–to determine whether they can afford representation. Furthermore, the requirement also implicitly recognizes that the justice gap exists not just for those with incomes close to the Federal Poverty Level, but in some areas, more than triple or quadruple that level. Thus, by associating the low-income requirement with the ability to afford an attorney requirement, FLA is increasing access to people who might have otherwise self-selected out of the process and is focusing its help on those with unmet legal needs.  

Second, this proposed qualifications checklist will not replace the entire user agreement; it will simply be at the top of the existing user agreement and will be the focus of the page. While the other information on the user agreement page is also important (what FLA offers, Google Translate, about the lawyers, rules lawyers must abide by, and confidentiality), it is arguably more imperative that individuals know whether they qualify to use the service in the first place. Hence, the qualifications should be in larger print and easier to read. For users interested in learning more about the service before deciding to proceed, they can read the remaining information on the webpage. 

Touchpoint: Submitting the Question

Question Intake

Once site visitors determine that they qualify for the service and want to submit a question, we propose a redesigned question intake process that immediately filters users into legal issue areas before they ask their specific question. We understand that this is a more substantive change that might be more resource-intensive than other suggestions. But this redesign addresses two important concerns. First, it addresses the FLA-side concern about not having the capacity to respond to all the inquiries that attorneys currently receive promptly and efficiently. Increasing outreach, particularly to communities of color, will only increase the volume of inquiries that FLA receives.

Second, from the users’ perspective, many of our interviewees found the question intake process to be cumbersome and, at times, overwhelming before they were even prompted to ask their question.

Our recommendation is modeled off of digital customer service platforms, particularly Amazon customer service. Amazon minimizes the burden on its customer service staff without compromising quality by filtering inquiries through prompted questions and different site branches that direct users to pre-prepared, common responses. We have a similar vision for FLA (see below).  

First proposal: triage questions by high-level categories

Second part: Have people select from common languages in the broad category

Through this system, users who have common legal questions can receive immediate, verified information without having to wait for a response from an attorney. Many of our interviewees indicated the importance of receiving quality information quickly. And for users whose questions are not answered through the pre-prepared question-answer system, they can proceed to ask their question via the question submission process. In implementing this plan, we suggest FLA’s pro bono attorneys work with state and local legal aid groups to draft responses to common legal inquiries. Such responses can be informational in nature, which would not violate rules regarding the restrictions to providing formal legal advice.

This new design is ultimately more efficient and cost-effective for FLA. While it requires an upfront investment to build out the new site features and populate the site with legal information, in the long run, it will minimize the number of repeat questions and overall volume of inquiries that have to be responded to by a human attorney.

Touchpoint: Receiving the Response 

Selecting Response Method

 In the spirit of continuing to streamline the user experience and to eliminate unnecessary barriers to access, we propose that FLA consider giving users a say in how they wish to receive answers to their questions. As one of our interviewees noted, users should not have to create an account or log into Google merely to ask a question. In some of the stakeholder and user interviews we conducted, people expressed frustration with having to log back in to retrieve their answer. To some users, a password-protected answer on a separate platform seemed unnecessary and inconvenient. 

As it currently stands, requiring users to log back in to their FLA account to retrieve their answer could create a number of issues. First, some users may simply never log back in. Second, having a password-protected service may contribute to this sense that FLA is gatekeeping access to legal advice. Many individuals find the FLA website intimidating; it is  a “lawyer” website, where users are instructed not to lie, and to attest to the fact that they qualify for the service. Requiring users to formally create an account may dissuade them from using the service entirely. 

As modeled in the above image, we propose that users be given the option to receive their answer by email, text, or via the FLA portal (as is the current practice). Many people rely heavily on their email and, especially for younger generations, their text messages. Allowing users to choose their desired response method ensures that legal answers are not just received but seen by those asking the questions. Moreover, adding the option to be contacted by email or by text message does not require FLA to eliminate password protection entirely.

FLA could still require all users to create an account before asking a question. And some users may still elect to receive their answer via their FLA account. Preserving the option for a password-protected response would cater to users who have particular concerns regarding the privacy of their communications with FLA. In particular, people facing any kind of violence or abuse in the home may be particularly concerned that a response to their question be kept private. 

A more radical change would be to eliminate password protection entirely, so that no one is required to create an account. However, we do not recommend this practice as such an alteration may compromise the privacy of those who may not want or cannot risk having their communications with FLA be so easily viewable by others. 

Auto-Reply Confirmation

After users submit their question, ABA Free Legal Answers should send users a confirmation message. This confirmation message serves two main purposes: (1) it builds trust and transparency; and (2) it empowers users with information about other resources while they wait to receive a response.

In this confirmation message, we also recommend including an estimated time frame in to help guide users’ next steps. Users will know when to expect a reply and whether to look for other resources for support and assistance in the meantime. Even if the wait time is lengthy, FLA should still include the wait time information so as not to mislead users. It is critical to provide accurate information to users so that they maintain realistic expectations with the service. Additionally, if the data does show that FLA has a longer response time, this would also be useful information for the ABA and its stakeholders to have.

However, if there is too much variability in the estimated response times to provide users with an accurate time frame, FLA can send a more general confirmation message. The message can simply acknowledge that the question was received and assure users that a response will arrive shortly. Even without an estimated time frame, the confirmation message will provide assurance to users that the FLA service is a legitimate and active service. In addition to providing assurance to users that a response is on its way, the confirmation message can also serve to educate users about other resources they can turn to while they wait for a response. Such resources can include other ABA resources, FLA’s Legal FAQs page (described in detail below), state legal aid groups, and court self-help websites. The links to the latter two resources—the state legal aid pages and court self-help resources—would not only help to facilitate a more unified ecosystem of legal help resources, but also potentially encourage the more reluctant state legal aid groups to agree to partner with FLA in their state.

By including links to external organizations, such external groups are more likely to see FLA as a supplemental resource in the broader legal aid ecosystem.   

In addition to providing a legal question and answer service, we recommend FLA consider consolidating frequently asked questions across a range of civil legal topics and publishing the answers to these questions in a separate tab on the website titled, “Legal FAQs.” The purpose of this feature is to provide users with a directory of pre-answered questions so individuals can get their basic questions answered in one place. The directory may answer all of a user’s questions, or it may be a starting point so a user can gain more familiarity with a legal issue they are dealing with. On the attorney side, we anticipate volunteer attorneys will have fewer repeat questions to answer after the implementation of Legal FAQs. Attorneys will then be able to devote more time and energy to answering questions that are not already answered in the directory or questions that may include more unique personal circumstances that require greater individualized attention.

It would also help to reduce the response time for attorney answers to questions, thus making FLA a more reliable and timely service. The lower volume of questions may also give volunteer attorneys greater capacity to answer more than three questions per year for each user, which is a current limitation to the service. In addition to providing information about common civil legal issues, the Legal FAQs could also include resources to help individuals prepare for court. A salient issue for pro se litigants surrounds the mechanics behind showing up to court. In a system made by and for attorneys, non-lawyer pro se litigants could feel scared and uncomfortable navigating the legal system.

Thus, providing those unfamiliar with court “culture” with practical advice on how to conduct oneself in court, how to address a judge, and how to prepare beforehand can help equip people with information they need to advocate for themselves in court successfully. Post-Submission Survey 
When a user has received an answer from an attorney, either via email, text, or through the FLA portal, we recommend FLA include a post-submission survey. FLA has used Qualtrics in the past to house a similar survey, but that survey is currently inactive. We recommend reinstituting the survey to ask users questions such as how they heard about FLA, how satisfied they are with the service they received, and how likely they are to recommend the service to a friend or family with a legal problem. To ensure that responses are as unbiased as possible, the brief survey should be anonymous. Collecting survey data would not only help FLA improve the quality of its services, but also provide data on how best to improve its outreach strategy.  

Data-Driven Next Steps 

In addition to our suggestions on how to improve the user-experience on FLA, we proposed three data-driven next steps that relate to outreach and resource allocation as FLA seeks to update its service. 

Outreach to Specific Demographic Groups

Based on our research, which included discussions with community members as well as advocates who work in the access-to-justice space, we gathered that among the most trusted sources of information for many demographic groups, including groups who speak another language aside from English or who are people of color, are local community organizations and groups. Thus, we suggest having FLA’s local state administrators contact and partner with local organizations that are highly respected by local communities to drive traffic to the website. Religious organizations, places of worship, civil rights groups, churches, libraries, and community centers are just some examples of the kinds of organizations that could help improve FLA’s reach across diverse communities

Collecting More Data

We also suggest that FLA record a smaller sample of users (possibly two to three users) as they navigate the website. This video recording of users perusing the site would provide a window into how users are currently experiencing the website, what problems they are encountering, and how much information they are retaining. Collecting this qualitative data would provide the ABA with critical information about what is working on the site and what is not. This could also help inform next steps as FLA decides to redesign the website. With permission from the users, the recording could also be shared with the ABA’s stakeholders to advocate for greater resources to revamp the website. 

Measuring Drop-Off Rates 

Lastly, we suggest that FLA use Google Analytics to track drop-off rates for each page of the website. Google Analytics is a free and easy way for businesses to track how many visits each page on their website gets and at what point do users start to click out of the website. For instance, Google Analytics could show what percentage of site visitors make it to FLA’s user agreement and how many do not continue on to the next step. Such data would allow FLA to diagnose problems on the website and optimize users’ experience. 

Categories
Blog Design Reviews

Virginia’s Eviction Legal Helpline website review and strategy

This report is by a team of Stanford University students in the Spring 2023 Justice By Design class.

The student team was Gabrielle Braxton, Shirley Frame, Whit Froehlich, and Lavi Sundar.

The students conducted a design review and user interviews in partnership with the Eviction Legal Helpline team. The goal of the class was to identify actionable technology, design, and policy interventions to help more people find good quality legal help online.

Introduction

The Virginia Poverty Law Center’s Eviction Legal Helpline website functions as a portal for Virginia residents facing eviction to access legal information, legal aid services, and its own helpline service. This situates it within an ecosystem of various forms of renter assistance, giving it an audience that includes tenants in different vulnerable circumstances as well as attorneys and non-attorneys who volunteer their time to operate the legal helpline itself. Given these audiences, the website presents visitors with options for self-definition so that users can find what they are looking for. At the same time, particularly for tenants, its layout and content serve to establish the credibility and trust necessary to successfully convey the information and connections it has to offer.

With the gracious agreement of VPLC, we had the opportunity to evaluate the Eviction Legal Helpline website, conduct user research, and consult with experts to formulate recommendations for how to improve its format and functionality. In the sections that follow, we detail several suggestions for approaches to different aspects of the design and rollout for an updated website, including general principles, potential tools to use, and specific examples of implementations on other websites that illustrate effective applications of good website performance. These are of course only possibilities, based on the limited scope of our project, and would be undertaken subject to the priorities and resources available for such work.

Over the Spring Quarter, we built a foundation of understanding about the nature and availability of online legal help resources, then conducted interviews with model users to investigate how they navigated the website and engaged with its content. Through these interviews, we found that the site is seen as inviting and useful, but that several of its resources went unnoticed or raised additional questions. Some interviewees identified specific aspects of navigating the website that would benefit from design adjustments.

We also analyzed the site’s web presence, drawing on our research as informed by guidance from our experts. Although the site appears highly ranked in search engine results, search engine optimization is a continuous project, and comparison websites demonstrate areas for improvement in the Eviction Legal Helpline’s rankings and reputation. Furthermore, while most prospective users will likely use search engines to find the site, the search-engine status may not fully reflect the effectiveness of the site in reaching its intended audience if they are not utilizing expected search approaches, or not using a search engine to seek legal help. Nonetheless, search engine optimization remains one of the most impactful means of ensuring users find the site and take advantage of its offerings.

Please find here the link to our final presentation, or read on for our key recommendations.


Design & Content Proposals

Plain Language

Increasing the simplicity of the language on the site will make it more accessible to everyone, including individuals with little time, learning disabilities, or limited English ability. With that in mind, we recommend writing the entire website in 6th-8th grade language. For example, this could look like updating the questionnaire with a simpler phrasing like “Why might your landlord evict you?” rather than “What is the reason your landlord might evict you?”

Simple, Straightforward Messaging

To further simplify language, reduce large chunks of texts on the homepage.

Start with basic information that would be attractive and accessible to someone with a scarcity mindset. Be welcoming and empowering: “Don’t leave your house. Let’s get you started.” Signal that there are ways to deal with this legal issue and invite users in.

Translation technology & interfaces

Moving onto translation features, the website currently presents two pages for information, one in English, “Tenants,” and one in Spanish, “Inquilinos.” This format could be confusing and redundant to users who don’t need to see the page twice—they just need the website in the language they are most familiar with. This could be streamlined with a centralized translation button. There are several models for how to implement this.

These translation and floating language switcher options are possible on WordPress with Translate Press.

Example 1: Add a translation button at the top of the page. (Website linked)

This button is easy to access and attention-grabbing, and translates the entire website to Spanish.

Example 2: Add a floating language switcher.

Our favorite option, this button is convenient and easy to see, always floating at the bottom of the page as you navigate. It stands out without obstructing the content. This particular website uses Google Translate, which reduces the manual translation labor.


Example 3: Add a drop-down menu at the top.  This button format provides lots of options and is easy to navigate. Because it is stuck to the top of the page, it is potentially a little more difficult to identify but still a great option.





Mobile-First Streamlining & Navigation

Many of the users we interviewed primarily use their phones to search for legal information, especially right after encountering a legal issue. Therefore, it is important to update and streamline the website to make it easier to navigate on mobile devices. Some current challenges of the website’s mobile format are the amount of scrolling required to view photos and text boxes, where there is extra space and text, and the sidebar, which offers limited information.

There are also some redundancies in the information offered that make navigating on a phone more difficult.

For example, on the “Tenants” homepage, several buttons and links repeat themselves, and it’s not easy to scan and understand the information. It would be useful to consolidate identical resources in a single location or prompt, such as a large “Start” button, which would then direct users to the questionnaire and, subsequently, the information they need.

Simplified example:
:

Another option is to simplify buttons by eliminating extra text. Simplified example

We recommend the following websites with great mobile formats as models and inspiration.

The Legal Aid Society is a great example that facilitates easy scanning with large font and bolding. It also has welcoming and well-formatted photos and a comprehensive drop-down menu for easy navigation to information.


One way to limit potential overwhelm is exemplified by the Ohio Legal Help website, which relies on boxes and buttons more than a sidebar with links. There are a lot of resources & information, but the site uses icons and accessible fonts to facilitate easy scanning and navigation. One downside of this website to consider is the large number of options and buttons which could be overwhelming. In addition, there is still a large amount of text on some pages.

Considering the models from the Legal Aid Society and Ohio Legal Help, we conclude that a balance must be struck when designing mobile formats between providing detailed information and ensuring accessibility. Both the side bar and the centralized homepage with lots of buttons are good options, but the organization must weigh the pros and cons.

Redesign & rebrand the triage Decision Tree Quiz

The quiz feature on the website currently uses a multiple-choice format that may be confusing to some users. The many options may be overwhelming. During our research, we found that some users were not sure which category they belonged in, or felt that they could fit into more than one category.

This sometimes created a sense that they might have chosen the “wrong” option and may miss out on important information that could apply to them.

We suggest remodeling the quiz to a binary format to simplify the amount of information contained in each question to ensure that users arrive at the most pertinent and accurate information for their particular situation. This will also increase the user’s confidence in the information they ultimately receive.

One-Page Legal Process Summary

During our research we found that one of the most popular features on the website is the one-page summary of the eviction process in Virginia. Users liked the flowchart format and felt the information in this summary would be extremely helpful to them if they were facing eviction or experiencing housing issues.

However, most users did not find this resource without being directed to it, since the link is quite small and located near the bottom of the “Tenants” page. We recommend featuring this resource more prominently on the site. We suggest including the flowchart as an image directly on the site (rather than linking to a PDF) and presenting it to users along with the information they receive at the end of the quiz.

Having a “birds-eye view” of the entire eviction process can reassure a user who is uncertain and anxious about their housing situation, deepening their understanding of their legal rights and giving them greater agency to advocate for themselves throughout the eviction process. It may also be helpful to update the flowchart to break down the steps into groups and orient the user to key events (pre-court/post-court, service of documents, etc.).

Outreach & Discovery of the site

Having discussed formatting modifications to increase accessibility and navigability, attention must also be paid to methods of outreach. More specifically, we recommend undertaking SEO methodology in order to maximize website search traffic from interested users. Semrush, an American company that specializes in generating and distilling search engine analytics, was the primary tool used to furnish the information that we have compiled in the following portion of this report.

Authority Score

The first metric used to pinpoint areas of improvement was that of the authority score, visualized here in a polar chart.

The two lowest metrics here are ‘organic traffic’, which is low, and ‘link power,’ which also is concerning.

The ‘link power’ factor is primarily a measure of the number of relevant backlinks, i.e. the amount of other credible websites that link back to VPLC’s Eviction Helpline website. A higher number of backlinks to the site would increase the site’s ranking and placement on a search results page, thus boosting user discovery and engagement.

Upon generating a list of competitor sites with similar backlink profiles to the Eviction Helpline site, the only competitor to appear was virginiahousing.com.

This was thus the best option to serve as a benchmark. We compared what domains refer to this site, to see what backlinks & referrals the site is missing out on.


As the above report makes clear, several high-quality websites with significant user traffic – such as Spotify, Amazon, Indeed, and US News – contain links to virginiahousing.com but not to evictionhelpline.org.

Backlinks & referral strategy

Thus, we recommend that these particular sites be targeted, to the extent and capacity possible, to advertise the site or disseminate related information.
The term “top-level domain” describes the ending portion of a URL, such as dot com, dot gov, etc. The top-level domain report Semrush generated, as shown below, underlines that the majority of websites that backlinked to the competitor were dot com websites.

Emphasizing the dot gov and dot edu websites when choosing referral domains may be a useful strategy to take advantage of the increased credibility that customers tacitly assign to these top-level domains.

Keyword boosting

With the ‘link power’ factor addressed, our next recommendation aims to improve organic traffic through localized keyword boosting.

Although not all of the following keywords are necessarily applicable to the Eviction Helpline, emphasizing or increasing the frequency of the relevant ones would also serve to increase engagement with users. The website’s page titles, section headings, and text can all use the keywords that people are searching for. This will help the search engines know that this website matches people’s needs.

Below is an example of a keyword report the team could utilize, to find keywords that might apply to the website’s content.

Leases, Notices, and other official referrals

Another way to boost outreach is through court/government referrals and lease disclosures.

The Eviction Helpline is already featured on some government websites, such as va.gov and the Fairfax County website. However, some of the information is outdated.

We recommend updating the information on these websites to reflect the current nature of the service and making sure that any links to the Eviction Helpline lead to the correct site (these “backlinks” will also help increase SEO authority). It may also be helpful to reach out to other local websites, such as city and county websites, to include information on the Eviction Helpline.

This will boost community awareness and give VPLC greater visibility.

Court and Government Agency Referrals

Other places to feature the Eviction Helpline include Virginia’s court self-help website and the Virginia Statement of Tenant Rights and Responsibilities.

Can the Helpline be featured prominently on this official document, with the website URL, phone, and QR code?

These community websites/resources present opportunities to “triage” target users to the Eviction Helpline site. This could potentially be automatic based on the information users input (e.g., income level, issue, etc.).

 


Summary

The Eviction Legal Helpline is an important resource for tenants in Virginia facing eviction and a valuable project of the Virginia Poverty Law Center.

As a public-facing website, this service must balance its function as a repository of the available information and connections with visitor engagement in order to build the trust and authority to provide them. At the same time, it is important to avoid overwhelming or confusing users with misdirected or excessive content.

In order to achieve these at-times competing objectives, the website can be improved by some adjustments, depending on available resources and interest.

Following our own learning and research as part of the Justice By Design policy lab course at Stanford Law School, we have formulated the above recommendations to aid in this effort. We hope that they may help in upgrading the site to ensure that it continues to provide valuable aid to its vulnerable target populations, and that the Helpline can maintain and expand its reach.

Categories
Blog SEO and discovery

Backlink Strategy for Legal Help Websites

Who links to your legal help website?

The more other websites that link to you — especially government or educational ones — the higher your website will climb on search results.

This is called a ‘backlink’ strategy. Search engines look to the web of who is linking to who to determine which sites are valuable, relevant, and authoritative. If a website ending with a domain like .gov, .org, or .edu is linking to a legal help website, these ‘backlinks’ make search engines more confident that your site is worthwhile & authoritative.

To that end, we encourage legal help website administrators to spend some time crafting a local Backlinks Initiative, to get more robust links among authoritative government and educational institutions.

Our cohort members had advice on how to make an effective backlinks strategy in the public interest space:

Backlinks Strategy for Legal Help Website

  1. Identify Your Regional Ecosystem of Legal-Related Websites. Who are the websites that are helping people with issues related to legal and court issues? You can map who is providing services generally, and also run some Google Searches to see what is showing up for people online. Some of the common actors in a local ecosystem will be:
    • Governor’s office
    • Mayor and/or City Council
    • Attorney General, particularly the consumer protection division
    • Legal aid providers
    • Court main website
    • Court self-help/ ‘for the public’ website
    • Housing agency
    • Consumer protection agency/division
    • County law librarians
    • Law school librarians
    • Law school clinics
    • Local news outlets
  2. Reach out to the Leaders & Web Team for each of these organizations. This could be an email or a call to the organization, in order to
    • Introduce your legal help website & your team
    • Ask if their team might be willing to link to your overall website, or even to specific pages within your website that might help their websites’ visitors
    • Offer a link exchange, where you would post their website for your visitors who might benefit from their services
    • Offer pre-formatted text or code that they can use right away. You can borrow from the text our Lab has already written that lists out & explains the legal help websites in each state.

Hopefully the teams will be responsive, and you can work with them to get the links set up. Ideally, they will not just be posting a link to your site, but also including text to describe it & its value to their visitors.

Here are some of the strategies & experiences that our cohort members reflected on about backlinks:

Do it early, but also refresh later on: Many websites do a backlink outreach campaign when they just launch. But it’s important to refresh both your landscape analysis & your outreach to get more links every year. Check in on where else people in your jurisdiction might be visiting, and approach these sites to link to you.

Be aware that some links may cause trouble: Your website might do a link exchange, where you are linking out to the organizations that link to you. Usually this is a net positive, when it comes to increasing your apparent authority to search engines. Sometimes, though, court or legal aid websites may have security flaws — like invalid security protocols — that makes search engines distrust them. If your site is connected to this site, the search engine might lower your rank. You can try to reach out to these sites to help them improve their security features, so this problem is resolved.

If you update your site, be sure to keep your previous URLs. If you are going through a site overhaul, especially when you are transitioning to new URLs for your content, you don’t want to lose out on all the incoming links you may already have. There might be lots of sites that are linking to your old URLs. That both drives users to your site, and it increases your authority to search engines. If you keep your old URLs intact, but then redirect through 301s to the new content/URL, you can preserve this authority & ensure that the old links don’t break.

You can also ask other sites to include your logo, and can give them pre-made code to help them do this easily.

Some possible partners, especially courts, might ask to see if your website is ‘unbiased’ and has resources for both sides of a conflict. They may only be willing to link to your site if it can demonstrate this lack of bias. Not all courts will have this requirement, but it is worth preparing for.

Getting links up can be slow, but if you are persistent, you can find the right person on the team who controls the website & then work with them to get it online.

Keep track on your data analytics! Can you confirm when the backlinks were put onto the site? Watch to see if your visitors or rank change in Google Analytics or Search Console. Then you can communicate back improvements to the partners, reaffirming the value of the backlinks & also decide where else you might do more links.

Prepare the right links for the right partner. For some backlink partners, you might just want their visitors to come to your homepage. This might be true for someone coming from a library or a general legal aid page. But if the backlink partner might have someone with a particular issue — like it’s likely the visitor has a debt, employment, landlord-tenant, or family law issue — you can ask if the partner will link to specific guides, FAQs, or media for this issue. That way the link could be more relevant to the visitor — so they go straight to the most helpful resource without having to navigate your site.

Categories
Blog Design Reviews

Designing an Eviction Help court website

Our team at Stanford Legal Design Lab collaborated with a court, help center team, and community stakeholders in Cincinnati to build an eviction help webpage.

In this video, you can see a walkthrough of the choices we made to make the site as successful as possible in being discoverable, engaging, and usable to tenants and landlords.

It walks through our homepage design choices, and flags some of the important things other legal aid and court partners can do on their websites:

  • Make a clear brand and call to action that is about empowerment
  • Flag your jurisdiction clearly and often
  • Provide distinct pathways & options for the different parties in the case
  • Link to phone numbers and websites of financial assistance, legal assistance, and others who can help with the problem that’s related to the website
  • Give links to people who can help a person who decides they don’t actually want to DIY this legal process
Categories
Blog

Strategies for better language access in legal help

Margaret Hagan, May 13, 2022

How can more legal help providers get more of their information & guidance into more languages?

There is a giant language access problem in legal services. So many people who need help have issues with Limited English Proficiency (LEP). Ideally, people with LEP would have equal access to legal help articles, guides, FAQs, and services in their own native languages.

But there is not enough funding, staffing, and capacity to provide robust information & services in all languages needed. Especially since each jurisdiction or organization is having to do language access on their own — it becomes a huge budget & capacity issue.

A 2013 report for the Legal Services Corporation, “Can Translation Software Help Legal Services Agencies Deliver Legal Information More Effectively in Foreign Languages and Plain English?” by Jeff Hogue & Anna Hineline (pdf at link), outlines different strategies that legal aid groups can use to increase the capacity & accuracy of language access efforts.

© Jeff Hogue and Anna Hineline, page 5 of report

They outline various tech strategies that could increase this capacity to serve in multiple languages:

  1. Machine Translation (like a variation of Google Translate or Microsoft Translate), in which a computer program is receiving the text, and proposing the translation. There can also be human review of the Machine Translate.
  2. Human Translation, in which a person is proposing the translation based on their knowledge of language & the situation. This is the traditional way that language access is done. An organization hires a translation firm or interpreter to provide customized, one-off translations.
  3. Translation Memory, in which people record their translations into a database, and then when there is a new text to be translated — they draw on this existing database for the translation. This database could be private (held by a private company or group of translators, and thus cost money to access) or open-source (held by the community and shared without cost).

This third category — of a shared database of translations and glossaries — could be a powerful solution to get to scaled, accurate language access. What if legal aid groups & legal help websites shared their multi-lingual (and plain language) translations of paragraphs, sentences, phrases, and words?

If there was a collective, open-source effort to create a Translation Memory database, this could spread the costs out among many groups. Instead of each group translating their content, they could share their past translations and allow other groups to draw from this.

This can also avoid the potential harms of a machine translation solution. In that setup, the providers are hoping that the machine (and its algorithms) can provide accurate & understandable translations. They might have a human to help review this. But the Translation Memory approach prioritizes the expert human translation from the start and then uses technology to make that approved, hand-crafted translation more accessible and replicable.

The authors of the report highlight that this shared Translation Memory approach could be valuable but costly. Here are some of their recommendations:

“The amount of time and effort that needs to be put into developing and maintaining a high-quality glossary and translation memory is non-trivial. We recommend that the Legal Services Corporation convene a group of leaders from legal services providers, plain language experts, and court leaders to adopt or discard this approach.” (page 23 of report)

They also recommend gathering a similar group of stakeholders to explore what is ethically & technically possible with combining machine translation with human review or specialized legal glossaries. Could there be an effective way to build on top of Google Translate or Microsoft Translate? It would be important to have a group of stakeholders and expert reviewers decide if this is possible and ethical.

For either a Translation Memory or Machine Translate + Human Review approach, having a shared database of glossaries is a key step. Our team at Legal Design Lab has started gathering glossaries that already exist, to start building an open-source database of legal help-oriented translations.

Please feel free to write or share if you want to work on this project with us! We hope to push language access forward with this infrastructure work, that can lay the groundwork for more accessible and scalable legal help efforts.

Categories
Blog

Administrative Burdens & Citizen Experience Design

Margaret Hagan, Jun 28, 2021

How can we create government programs that people actually can use?

I have been working with my Legal Design Lab and the National League of Cities to improve people’s access to housing justice and eviction prevention resources.

Especially with COVID-19 hardships, there are more renters at risk of being evicted. They’re behind on rent, they’re in an unstable economy, and they need help dealing with back-rent, fees, court cases, and the threat of homelessness.

City, state, and federal government agencies have responded with new programs. There are mediation services, legal counsel programs, rental assistance funds, navigator services, and other things — often called an ‘eviction diversion’ program.

This work has brought up the question of burdens. Especially for tenants, who are having financial hardships, many of these eviction diversion programs require tenants to do lots of things to get access to benefits.

To get rental relief, for example, it means a tenant must fill in lengthy applications, gather documents, following months-long procedures, negotiate with their landlord, and figure out eligibility formulas.

This goes back to the issue: how do we design the programs in practice — not just the overarching policies? The devil is in the details. The process often is a punishment. How do we roll out relief programs that actually achieve their intent of keeping people housed, avoiding adversarial court proceedings, and stopping harmful scarlet ‘E’ eviction judgments on people’s records? How do we make sure the programs stop spirals into poverty?

This takes us back to the question about administrative burdens. If a policy is rolled out in a high-burden way, that makes it difficult to: find out about the program, sign up for it, and follow through on it. Then many times the policy goal will be undermined. People won’t be able to use the benefit, they won’t be able to exercise their rights, and the bad outcome is going to happen anyway.

One key book to have on the table, then is the wonderful volume, Administrative Burden: Policymaking by Other Means, by Pamela Herd and Donald Moynihan.

Below are some key points to take away from the Administrative Burden book— though I recommend you get a copy for yourself to dive into case studies of various government and state benefits programs, and how they’ve grappled with the politics and administration of administrative burdens.

We need to focus on citizens’ experience of government policies & programs.

The book points out that policy-making often focuses too much on the policies in the abstract and does not focus on their actual administration. There’s too much focus on the policyholder or the policymaker, and not so much on the citizen’s experience

More burdens are faced by those who have fewer resources to manage and overcome them. For many Americans, the experience of government is the experience of burden (see p. 7 of their book).

There are 3 main categories of burdens to track: Learning, Psychological, and Compliance Burdens.

We can measure how high- or low-burden a program is by looking at 3 components of a citizen’s experience:

Learning costs:

  • Time to learn about the program
  • Time to figure out if you’re eligible for it
  • Time to figure out what benefits you’d actually get
  • Time to access the program
  • Time to determine what conditions you need to satisfy to get it

Compliance costs:

  • How hard it is to assemble documents to prove you’re eligible, or what you should get
  • Financial and transactional costs to get services to help you get through the application — like lawyers or navigators
  • Travel costs to show up for interviews, file documents, get other supporting documents or fingerprints
  • Financial costs of fees to apply or get documents
  • Transactional costs to reply to communications from the program, meet deadlines, clarify requests

Psychological costs:

  • Overcoming stigma or embarrassment of using a service
  • Losing autonomy and privacy when opening one’s life to administrators evaluating them
  • Frustration of dealing with repetitive, unjust, and unnecessary procedures
  • Stress about the uncertainty of whether one can make it through the process
  • Sense of procedural injustice, of not having a transparent, respectful, and fair procedure

Some of these costs can be measured objectively, by gathering data about financial costs, time costs, and other quantitative measures. Others can be measured through surveys, interviews, and other design evaluations of people’s experiences.

Burdens matter to people’s use of government programs.

Burdens matter a lot. As policies and programs put more burdens on people, this affects whether people can actually get the benefits and rights that are at the heart of these policies.

And design matters to burden. This is a matter of user experience, good design, and community involvement. Good, community-centered, and creative design can shift burdens away from citizens and on to the government. How can we shift burdens to those with more resources to bear them? That’s most often away from citizens individually.

Design can help us measure burdens as we are creating new programs and services, and then evaluate them in their pilot stages. Are people being frustrated? Are they dropping out from onerous tasks? Are they becoming alienated from government services because of how burdensome a program is? We can use design techniques of user testing, UX evaluations, and human-centered evaluation to measure these burdens and create new strategies to repair them.

Every public service should have good citizen experience (and burden reduction) at its core.

As groups are making and evaluating new programs, they should be aware of citizens’ experiences and burdens. This means having these principles at the core of their work:

  • The program should be designed to be simple
  • Processes should be as accessible as possible
  • The program should be respectful of the people they encounter, and dignity should be at the core

Design can again help here. This can be done through User Personas, User Journey Maps, trackers of where people are failing or falling off, surveys about stress and procedural justice, and measurements of wait times and other objective measures of burdens.

We can create replicable strategies to lessen burdens and improve citizens’ experience.

How do we make Administrative Burdens & Citizen’s Experience part of front-line policymaking and service delivery?

The book points to a few directions:

  • Training policy managers & on-the-ground administrators in the importance of the citizens’ experience, these 3 kinds of burdens, and the importance of good service design. There also needs to be explicit training in equity & burdens — about whether people from certain demographic groups are being asked for more evidence, put through more process, and asked to shoulder more burdens.
  • Instituting more testing of burdens before and after a program rolls out. If we measure it, we’ll optimize for it. This means gathering data on the learning, compliance, and psychological costs — -through intentional data-tracking, running of surveys, mapping of user experiences and drop-offs, etc.
  • Deploying burden-reducing strategies that can reduce these learning, compliance, and psychological costs. Some of these burden-reducing strategies include the following. Many of them draw on nudge/behavioral heuristics literature.

Burden-reducing strategies for public programs

  • Limit eligibility criteria. Cut out unnecessary or overly burdensome ‘means tests’ that make people prove they are eligible based on various income and financial assessments
  • Limit the amount of choices, reduce cognitive demands, and label what the most common choice is.
  • Invest in community-based outreach, to make it easier to find out about the program and learn other people’s stories of it.
  • Label and brand programs in positive terms, that reduce stigma, embarrassment, and moralizing.
  • Auto-enroll people, presume they are eligible and cut out tasks they must do to get access to the program.
  • Figure out which party is well-resourced, and pass high-burden tasks around document uploads and financial accounting to them.
  • Possibly do cross-overs between programs, integrating data from other programs so there’s no need to fill in forms with information the government already has, or upload new documentation.

Open Question: does lowering burdens on citizens mean trading off privacy?

One question to balance the burden discussion with is around people’s privacy, and control over their own data. Many strategies around lowering burdens mean collecting and connecting data together.

Especially if it is poorer people applying for these programs and needing ‘burden-reduction strategies’ — it’s likely to be their data that is being handed off between programs in order to make it easier and quicker to use a service.

Does reducing burdens lead to a reduction in privacy from the government? What is the balance between an all-knowing government, in which agencies are passing off information about a person back and forth — and an easy-to-use government that is significantly easier to access? There’s a huge need for design sessions, technical solutions, and policy work around this trade-off, of low-burden government services that also protect vulnerable people’s privacy.

Categories
Blog

What can legal learn from medical when it comes to ethical AI?

Margaret Hagan, Jan 27, 2021

Last night at my Public Interest Tech Case Studies class, our guest speaker was Dr. Tina Hernandez-Boussard of Stanford School of Medicine. She is a multi-hyphenate: doctor, epidemiologist, and researcher who works on how AI algorithms are being developed and deployed in health care. One of her lines of work is looking at whether the AI applications being deployed in medical work are not only technically robust, but also ethical, human-centered, and socially just.

The promise of AI in medicine is huge: if researchers, machine learning experts, and clinicians can draw on all of the past data of patients, symptoms, treatments, and outcomes — they may be able to offer better, quicker, more intelligent help to people who are struggling with illness.

Dr. Hernandez-Boussard identified 3 main tracks that medical professionals are using AI for. These may be analogous to legal tracks that are beginning, or may begin soon, for legal help & AI. They are:

  1. improving biomedical research (like in surfacing important findings from researchers, and links between studies),
  2. doing translational research (like in how the genome affects diseases and outcomes), and
  3. improving medical practice (like in how diseases are diagnosed, treatments are selected, patients are monitored, and risk models of disease/outcomes are built)

What potential is there for AI for Access to Justice?

This third track is perhaps the most exciting one for those of us focused on access to justice. What if we could better spot people’s problems (diagnose them), figure out what path of legal action is best for them (treat them, or have them treat themselves), and determine if their disputes are resolved (monitor them)?

Our Legal Design Lab work with Suffolk LIT Lab is already working the first thread, of spotting people’s problems through AI. We had collaborated on building Learned Hands, to train machine learning models to identify legal issues in people’s social media posts. That’s led to Suffolk LIT Lab’s SPOT classifier, that is getting increasingly accurate in spotting people’s issues from their sentences or paragraphs of text.

This medical scoping of clinical AI uses could inform future threads, aside from issue-spotting, for legal aid groups, courts, and other groups who serve the public:

  • Legal Treatment AI: Can we build tools that predict possible outcomes for a person who is facing a few different paths, regarding how to resolve their dispute or issue? Like a tenant who is having problems with a landlord making timely repairs: should they call an inspector, file for rent escrow, try to break their lease, use a dispute resolution platform, or do nothing? What would be the time, costs, and outcomes involved with those different paths? Many times people seek out others’ stories to get to those data-points. What has happened when other people take those steps? AI might be able to supplement these stories with more quantitative data about risks and predicted outcomes.
  • Problem Monitoring AI: Can we build tools that follow up on a person, after they have interacted with legal aid, courts, or other government institutions? Did their problem get resolved? Did it spiral into a bigger ball of problems?

The need for community design + ethical principles in AI development

That said, with the promise of these medical-inspired threads of legal help AI — Dr. Hernandez-Boussard warned of the importance of careful design of the AI’s purpose, data sources, and roll-out.

The danger is that AI-specialists develop new algorithms simply because it is possible to do so with available data sets. They may not think through whether this new tool (and the data it’s based on) is representative of the general population and its diverse demographics. They may not think whether clinicians would actually use this algorithm — whether it solves a real problem. And they may not think about unexpected harms or unequal benefits it might result in, for the patient.

This shows up with algorithms that detect heart failure in men, but don’t work at all for women. This is because the data the algorithms were trained on, is based on trials populated mainly by white men. Their symptoms for heart attacks are markedly different from women’s symptoms. So the model doesn’t detect women’s risks accurately, and may result in women not getting the prioritized care or appropriate treatment. A similar story is developing in regard to skin cancer screenings, in which the dataset training the AI is mainly from fair-skinned patients. Thus, the tool likely won’t be as effective for screening cancer in darker-skinned patients.

AI built on non-representative data, or rolled out with too much trust in its predictions, may result in poor care, bad outcomes for unrepresented groups, and less overall trust in the health system (and the AI).

Dr. Hernandez-Boussard is working on a better framework to think through the development of AI for clinical care.

  1. Stakeholder Involvement in scoping the AI project, setting standards, and limits
  2. Data cleaning, quality-checking, and pre-processing — to make sure the data is as recent, accurate, representative, secure, etc. as possible
  3. Development of tech and testing of its fairness — to make sure that it is making accurate predictions, especially across protected classes of gender, race, etc.
  4. Rolling out the AI to be transparent and usable — so that practitioners don’t over-rely on it, use it for problems it wasn’t meant to solve, and to make it comprehensible and usable to patients and their care teams

This involves early, deep stakeholder involvement. In this phase, there are critical discussions about whether AI is needed at all, what important questions it can answer, and whether clinicians and patients would actually use it on the ground.

The promise of co-design in AI development

This phase is where legal design could shine in AI/A2J work. Co-design, participatory design, and community-led design methods are meant for this type of broad stakeholder conversations, agenda-setting, and principles-setting.

Legal design is also essential in the fourth phase of development: how is the AI rolled out to people on the ground who should be using it to make decisions. Do they know its limits, its standards, and its sources? How can they be sure not to over-rely on it, yet still build trust in what it is able to do? And how do they help overwhelmed patients make sense of its predictions, risk scores, and lists of percentages and possible outcomes?

Are you interested in learning more about ethical AI in healthcare, and how it might be used in other fields like legal services? Check out these upcoming events and online courses:

Its summary: Artificial intelligence has the potential to transform healthcare, driving innovations, efficiencies, and improvements in patient care. But, this powerful technology also comes with a unique set of ethical and safety challenges. So, how can AI be integrated into healthcare in a way that maximizes its potential while also protecting patient safety and privacy?

In this session faculty from the Stanford AI in Healthcare specialization will discuss the challenges and opportunities involved in bringing AI into the clinic, safely and ethically, as well as its impact on the doctor-patient relationship. They will also outline a framework for analyzing the utility of machine learning models in healthcare and will describe how the US healthcare system impacts strategies for acquiring data to power machine learning algorithms.


This post was originally published on my blog Open Law Lab.

Categories
Blog

An Easy Guide to Creating User Friendly FAQs

Roda Nour and Katie Yoon, Nov 15, 2020

The COVID-10 pandemic has precipitated much national chaos and confusion with regard to the legal process of evictions. Several local governments and courts have been adjusting their policies to accommodate the current catastrophe, such as by placing temporary moratoriums on evictions or relaxing payment deadlines for tenants.

In response to this housing crisis, we have been working as part of the Stanford Legal Design Lab to build a Legal FAQs platform providing jurisdiction-specific eviction information on a local, state, and federal level. Our primary task as summer interns these past few months has involved achieving 50-state coverage on legal, easily understandable content for people facing eviction lawsuits.

Based on the skills and knowledge we have acquired while working on the eviction platform, we have written below an easy guide to creating user friendly FAQs.

1. USING SIMPLE LANGUAGE

One of the most important things to keep in mind when creating user friendly FAQs is your audience. Who are you writing this for? It’s very easy to lose ourselves in legalese or whatever technical jargon we’re familiar with, to assume that we’re all on the same page, and for our audience to be left behind. As silly and Elementary-esque as it sounds, sometimes it helps to read what you’re writing out loud. If it sounds too complicated or verbose, you know that you need to go back and edit to try to keep it simple, both in language and conceptually. Keeping the language basic and easy to understand ensures that your users, no matter their background, can always follow along.

2. KEEPING CONSISTENT WORDING

On this same line of thought, it’s important to have plain language consistency. One way to do this is by creating an overall general template that you can use for each portion of your FAQs, and then to fill in the pertinent information as you go through answering the specific questions. This way the wording will be consistent all throughout.

3. REVIEWING WITH PARTNER

Be sure to review your work. It’s a great idea to work in a team or with a partner; this way you can review each other’s work to make sure that you are both using plain language, and using a template ensures that you both are using consistent language. You can catch things your partner might have overlooked or missed and vice versa.

4. TRACKING SOURCES

Another tip for creating FAQs is to keep a list of sources as you conduct your own research. Our team worked on creating FAQs for all 50 states. Starting was very difficult — not knowing how to navigate all the legal codes and research out there made it slow going, but once you begin to recognize sources that you can use or even certain terms or phrases to search, it becomes much easier. And so keeping a list of the sources or phrases you use to research will help you as you curate FAQs.

5. REACHING OUT TO EXPERTS

It’s also extremely important to prevent people from finding the wrong information. You can do this by reaching out to volunteers, professionals in whatever field you’re creating FAQs for, and asking them to review your work and ensure the accuracy of your answers. In our case, if even one of our answers is incorrect, and people read our FAQs and think that they have more time to answer an eviction suit than they really do, it can have serious consequences. This is why it’s very important to verify your work as you go along.

FINAL WORDS:

We at the Legal Design Lab used this process to successfully develop eviction FAQs and anyone can follow this guide to create their own set of user-friendly FAQs. One of the biggest takeaways from this article is to be mindful of your audience — FAQs packed with jargon aren’t useful to anyone, so make sure to use simple language! Through these five easy steps, — using simple language, keeping wording consistent, reviewing your work, tracking your sources, and reaching out to experts, you’ll be able to create accessible and easy to understand FAQs while maintaining verified correct information.

Categories
Blog

Public Interest Project (PIP) Hub

Michael Swerdlow, Sep 23, 2020

In the era of COVID, public interest organizations from legal aid societies to public health departments have never been busier while many students have never had more free time. So, too, our civic need for effective social problem-solving has never been greater. What better time to launch a curated platform for social problem-solving and civic technology?

What if there was an online exchange and information-sharing forum, the Public Interest Project Hub (PIP Hub), that would enable students and public interest organizations to connect to share ideas and coordinate projects to address civic needs.

Public interest organizations or citizens with problems who understand public and organizational needs could add project proposals to the Hub. These ideas and proposals might range from a client intake system that facilitates real-time statistical analysis to a know-your-rights app that helps people understand regional laws and advocate for themselves, or a hundred other creative ways to address or mitigate collective problems. Students with relevant policy, design, and technological skills could then connect with the organization in a volunteer or contracted capacity to develop a project plan that leads to a research report, tech tool, website, or another outcome that helps scope or address the issue at hand.

PIP Hub, diagrammed

GitHub has been wildly successful in allowing people from across the world to collaborate on technical projects such as Bootstrap and JQuery; but, without guaranteed and actively managed student and community/organizational participants, similar spaces for public interest technology have not emerged. This is a shame as there are likely many front-line organizations who have unmet technical needs and many students who would be excited to support them in developing solutions.

Think of PIP Hub as a forum combining features from Google Drive and GitHub. The drive would be the external-facing system that students and organizations could use to connect with each other. It would contain a project idea intake form and a Google sheet listing open projects that students could either apply to or just start working on. It could also contain white papers describing organizational and community needs; completed student research on public policy and/or social problem-solving issues; and folders that link to completed or ongoing social problem projects on GitHub. The PIP Hub would facilitate the collaborative development of technical projects. It would also make it easy to build open-source projects available to a broad community of developers and users.

Students who have worked on policy/civic tech projects but who have since stepped away from their work could also benefit from the hub. Many project-based courses require students to research and prototype projects, but after the term ends most of those projects never move forward. The next year another group of bright-eyed students enters the same course only to repeat the cycle. Students who worked on policy proposals or civic tech projects could leave their projects in the drive and then allow students who take a similar course, or are just interested in the project, to pick up where they left off and move the project further toward real-world use. Public interest organizations could also view these projects, provide feedback, and write proposals for a group of students to build a tool based on a student prototype.

Some public interest organizations may hesitate to engage with PIP Hub, pointing out their need to own the data they create and control the technology they rely on. Yet, the counterpoint is a pernicious trend in civic technology with proprietary software that privatizes public data, prevents community members from understanding a technology’s impact, and creates barriers for widespread adoption. To ensure ethical and effective design, all work-products that emerge from the hub should be open source or licensed through Creative Commons. If necessary for their mission, public interest organizations should be able to request that work products be closed source.

In sum, the PIP Hub could facilitate an innovation ecosystem grounded in civic engagement that would connect public interest organizations with unmet needs to students who are seeking ways to develop their skills in policy research and/or civic technology. It could also connect students who have started policy/civic tech projects to others looking to carry them forward.

To ensure a successful innovation ecosystem, PIP Hub staff would need to perform several administrative and constructive functions. They would need to publicize and solicit engagement for the drive from public service organizations, students, and professors. They could also structure the terms on which the work would be done and facilitate project sustainability by providing students with either course credit or grant funding. Once projects are completed, the Hub could publicize the products to similar organizations that could benefit from their use. Staff could also connect students and organizations to professors or foundations who could advise or fund projects. Lastly, they could organize and index documents so that content is easily searchable. On the Google drive, this could mean imposing a standard format on the tracking sheet and grouping projects displayed for external observation by issue area to be most accessible for partner organizations. On Github this could mean managing permissions and ensuring that each project has a README file containing a comprehensive summary of the project. Financing for the PIP Hub could be provided by such groups as the Public Interest Technology University Network or the philanthropies behind it. University programs and departments could pay small dues to give their students access, thereby enabling cross-university partnerships and team experiences for their students. Public interest organizations could join as members. Corporations could sponsor the hub as donors. Throughout, the Hub would retain its independence and neutrality to enable civic organizations, community members, and students to partner freely on projects. It may also be advantageous to pilot several different hubs with different funding and administrative models based out of different research universities. After a trial period, the hubs could compare, adopt best practices, or merge into a unified system.

In short, the PIP Hub would contribute to a world where communities can join together in collective problem solving to find sustainable solutions to public problems. It would facilitate the distributed creation of high-quality research and technical tools that public interest organizations could use at low-no cost. It could help communities share information and find collective solutions to problems they identify. It could serve as a training ground for a generation of students interested in applying analytical and technical methods to societal problems.


Michael Swerdlow is a recent Stanford graduate and admit to Columbia Law School. If any organizations are interested in creating their own PIP Hub feel free to reach out. Contact: mswerd@stanford.edu

Categories
Blog SEO and discovery

Using Schema.org to Help People Find Legal Aid

by Nora Al Haider

(Originally published on Legal Design & Innovation medium publication)

Overview

This piece documents an intervention aimed at improving access to justice outreach. We used a technical intervention — standardized data markup on legal help websites — to improve people’s discovery of key, public-interest legal information.

Intended Goal: Our goal was to help more people find out key legal rights & services when they searched online for help. Search engines’ results pages are a key place where people find out about legal help. We wanted to raise the placement of local, public interest legal help sources on Google search results pages. Ideally, a person searching with a problem like ‘eviction help’ or ‘how to get a restraining order’ will see local legal aid groups that can help them resolve their problem, or legal guides from their jurisdiction that can build their legal capability

Our Intervention: We used website structured data markup, called Schema.org, to improve how search engines found & displayed legal aid websites to people searching online.

Our Team: Our team at Stanford Legal Design Lab partnered with local legal aid groups across the US, with support from the Pew Charitable Trusts & the Legal Services Corporation.

Schema Markup Intervention

In the early months of 2020, our team worked with stakeholders in Florida, Hawaii, New York, Alaska, and Idaho to implement an intervention that could improve the placement of legal aid and legal help websites in search results. We created schema markup for our stakeholder organizations’ websites, with the expectation that this structured data could help their public interest organizations show up higher on search result pages, and could match them with more appropriate users of search.

The non-profit Schema.org consortium of major web search providers, including Google, Bing, Yahoo, and Yandex, sets the standards for the markup, and also uses it with its search engines. Schema markup is a way that websites can communicate to search engine crawlers and bots. It is JSON code that is put on the back end of an individual website page, which indicates information about the organization that runs the page as well as the services and content that is on the page itself. It uses standard markup language, that all of the major search engine providers agree on, to communicate these key pieces of information to the search engines automatically.

Markup is often used by commercial businesses in their search engine optimization (SEO) strategies. In addition to other SEO techniques, like having more websites linked to your website, creating content that matches with common search terms, and having fast loading and mobile responsive pages, schema markup and other structured data techniques can help an organization get more prominence in people’s search results.

One of the main issues with Schema markup is the high barrier to knowing how to use it for a particular type of organization. The markup itself is a collection of terms that can be defined however any given webmaster or content developer wishes to. It is free to use, but takes a great deal of expertise to understand how to use it correctly for a certain type of organization, website, and set of services.

Steering group of stakeholders, on legal help markup protocols

For this reason, our stakeholder group focused on developing a standard protocol for legal help organizations who wanted to mark up their sites with schema markup. Over the course of six months, our lab team drafted and revised a standard way to use the markup for legal aid organizations, statewide legal help portals, and other legal information websites. We did this with constant feedback from these organizations’ webmasters, lawyers, and content developers. We held twice-monthly phone calls and Slack office hours, in which we presented our proposed markup protocol for these stakeholders’ revisions and approval. Their input helped us to understand:

  • What types of organizational details, services, and site content they wanted to make sure that people searching online should find, and what information they would rather not be shown publicly (or shared widely).
  • What the right ‘unit’ for sections of markup should be: like describing all services around the main organization, or describing the services that each office of the organization provides, or describing the services that each unit of the organization provides. We ended up with a ‘graph’ model of the legal schema: describing the main organization, then describing each service it provides, and linking them together using key ids that made it clear that the organization provided the particular services.
  • How to actually implement the schema on their websites and content management systems, to get the JSON code onto their homepages without disrupting their current codebase and SEO strategies.

Our markup covered the pieces of information about the organization, as well as its general legal aid or legal help services. In our first round of markup, we decided not to mark up events, one time services, or other pieces of information that were likely to change quickly. Our stakeholders warned us that they often don’t take down stale information on clinics or events, so we may inadvertently display events to users that no longer take place (like bi-weekly office hours on eviction help, that may have been halted several months ago). They explained that their events are often funding-contingent, so when a 12- or 18-month grant ends, the event service ends but they do not necessarily update the website with this information.

Instead of events or other information that changes frequently, we focused on the stable information about the website, like the organizational details, contact information, jurisdiction, areas of issue expertise, price range, and type of organization. This organization-level markup may have an effect on these organizations’ sites being displayed more prominently, potentially with a map, business hours, description, and contact details shown on the results page. It was not page-level markup, that would be more likely to have specific sections of content, FAQs, or step-by-step how-to’s appear on the results page. This first round of schema markups is available for public review.

An example of Schema.org code for Idaho Legal Services’ website and legal aid services

By Spring 2020, we had an agreed-upon set of terms and ways of laying out schema markup for legal help sites. Our Lab handcrafted markup for our partner organizations, following the protocol. We gave the seven legal help organizations the text files of the markup for them to apply on the back end of their main home page. Because each group had different content management systems, there was wide variation across our cohort surrounding how easy or hard it was to implement the markup on their website. Typically, markup is implemented by manually inserting it into the backend code of a website, like in homepage’s header code. In some cases, legal aid groups use other search engine-related apps on their content management systems that interfere with new markup being added. We had intended that all sites would implement it at the same time so that we could have regular check-ins to see any effects or watch for effects of possible external variables like new policies on unemployment or rental housing for COVID. But there were gaps of several weeks and months between the different sites’ schema implementation.

Expected impact of the markup intervention

Our expectation was that the schema intervention would increase the traffic of jurisdiction-correct visitors to the website. There have not been controlled studies of precisely how markup may increase overall traffic, or jurisdiction-correct traffic. It was unclear whether overall traffic volume would go up. Instead, the goal was to increase the jurisdiction-specific traffic of people from the state or the region that the organization serves. That can sometimes be seen through Google Analytics, which shows the state or region from which visitors are apparently located. (Though, it is important to note, that people using various browsing tools may obscure their location or prevent tracking, for the sake of privacy). We also expected to increase the number of visitors who were interested in the subject matter of the organization, as shown by the time they spent engaging with the site rather than ‘bouncing’ (or leaving quickly, apparently because the site did not match their intent). We wanted to find people whose search queries’ intent matched the services and information on the websites with the markup.

This first round of the schema intervention was also meant to be a learning experience in developing the intervention itself. As there is no official protocol on how to use the myriad schema markup terms to represent a given group — particularly a non-profit group (as most markup so far has been done by commercial groups, and examples are geared towards businesses) — this experiment was meant to to identify bugs, develop a common protocol, and begin to see how much of an effect on traffic structured data markup could have. As we created and implemented the markup, the goal was to learn 3 main things:

  1. How to automate the creation of markup so that more organizations could do it quickly if not automatically.
  2. How best to do more detailed markup, that would even better specify key terms about service, issue expertise, and jurisdiction, to improve how search engines showed information about these sites. In the first round, we kept a fairly short list of general, site-wide information to mark up, but in future rounds we hope to mark up more of the websites’ individual pages, guides, hotline, clinics, and other services they offer.
  3. Whether the hype around schema markup’s ability to improve site’s rankings translated into real impact. Could markup increase how often sites appeared in the top 10 or top 3 search results? Could it increase the number of jurisdiction-correct visitors, who would spend time on the site to make use of its resources? If it did not have substantial impact, this would change our strategies, to possibly focus on more engagement with policy leaders at technology companies, about how to improve public interest organizations placement, aside from structured data.

Once the seven organizations applied schema markup to their websites, we then used a combination of individual site analytics and post-markup search engine audit to analyze these three areas of inquiry. The search engine audit helps us to see what people are being shown by search engines, when they search for common legal problem terms. The site analytics help us to see who is finding their way to public interest legal help sites, what queries or referrals brought them to the sites, and how they behave when they are on the sites.

Website Analytics & Search Audit: What was different, what did the intervention show?

After implementing the Schema markup intervention on the seven websites, our team led an evaluation of what the sites’ Google Analytics showed in terms of impact before and after the markup. This is in addition to a post-markup audit of search engine results, to see if the public interest sites appeared more frequently or in higher positions after they had markup on their site.

Whereas the search engine audit informs us about what sites people are shown as likely relevant to their question, website analytics inform us about how many people end up coming to a website to find help for their problems. We use both measures to see how markup affects how search engines display listings of sites, and how users ultimately behave in choosing sites to visit and spending time on them.

Google Analytics provides an overview of website traffic. It includes reports and analytics on traffic sources, locations, demographics and behavior of the audience. We used the Google Analytics data to analyze the effects of the Schema markup intervention, and the impact of COVID-19 on the aforementioned markup intervention.

How soon does markup make a difference?

Generally, the effects of schema markup cannot immediately be evaluated after they’ve been implemented on a website. There are varying online reports, and no clear answers, but according to most informal message board threads, the effects of markups are visible on search engines and analytics pages anywhere between 1 week and 1 month. It may take a while because of the cycles on which search engines’ crawler bots search the Internet to index new sites, content, and markup. They may be delayed in ‘finding’ the markup, if they only visit the site once a week or once a month to look for updates.

In our project, the schema markups for the legal aid and service organizations were all implemented on their websites on different dates. In order to provide a valid evaluation, we had to ensure that enough time had lapsed between the implementation of the markup and the analysis.

Schema Implementation Records

Organization & Schema markup implementation date

Idaho Legal Aid: 5/6/2020

Legal Services of North Florida: 5/11/2020

Law Help New York: 5/29/2020

Legal Aid Hawaii: 6/2/2020

The above info provides an overview of the implementation dates. The analysis was conducted on the websites’ analytics at the end of August 2020.

How are the sites performing?

Two main sources were mined for data: Google Analytics and Google Search Console. It is important to distinguish these two tools. Google Analytics provides more insight about the visitors, page visits, and usage time of the webpage. It tracks and reports different segments of website traffic. Google Search Console, on the other hand, provides more insight on the organic traffic search results. Organic search traffic indicates the visitors that visit the website through search engines. We used both of these data sources to analyze the impact of the schema markups.

Measurement issues

Before delving into the results there are some measurement issues and important assumptions that may have impacted our results in visible ways. These issues will be discussed more fully in the following paragraphs.

  • Different implementation dates: The implementation date is different for each of the legal aid and service websites. Although we left the minimum required time (1 week) between each implementation and the start of the analysis, there are some unknown aspects regarding the detection of the markup by search engines. This might have created some variances in the analysis.
  • Missing data: There was the issue of missing data for some of the legal aid websites. Florida Law Help switched ownership in early 2020. Google Analytics was disabled when it was moved to a different platform, therefore there is missing data in the months of April, May and June 2020.
  • Correlation/Causation: Although some of the data suggests that there is a possible correlation between the schema markup and the several of the metrics we analyzed, this does not necessarily indicate causation.

Results: Did Schema improve visitors and quality matches?

We looked at four separate metrics to determine if and how schema markup changed the legal help websites’ discoverability: visitor count, traffic sources, session duration, and click-through rate. We recognize that particularly in a year with COVID-related upheavals, people may have changed their search behavior, and that this could confound our reliance on analytics. Ideally, in future years with fewer emergency events, there can be further study of website analytics before and after the implementation of markup or other SEO strategies.

Metric 1: Visitor Count

When thinking about impact, increasing the number of visitors to a website may seem to be the most important metric. Yet we know from our discussions with legal aid experts that they are most interested in increasing the number of ‘appropriate’ visitors — those from their service area and who have legal help queries — even if that means a decrease in overall visitors.

Visitor count can also be a problematic metric for measuring the effectiveness of a particular intervention, like schema markup. The visitor count metric will be affected by other variables aside from markup — like policy changes, economic downturns, natural disasters, and other events that may affect who is searching for legal issues. We can expect that a large swing in visitor count is likely due to an annual or special legal event. Theoretically, comparing the visitor count trend for 2020 with the trend for 2019 can help separate out the seasonal changes in visitor count from the effect of schema implementation, but because of the unique legal circumstances in 2020 (i.e. COVID lockdown, economic downturn, and related legal issues around housing, benefits, unemployment, family, schooling, etc.), such a direct comparison was not possible either.

Therefore we analyzed 3 other metrics, in addition to visitor count: traffic sources, session duration, and click-through rate.

Metric 2: Traffic Sources

We compared where sites were getting user traffic from. In particular, we compared direct traffic versus search traffic. Direct traffic means users are arriving at the website directly through a link or a bookmark, while search traffic means users are arriving through searching through Google. If an increase in visitors was caused by schema, we would see an increase in search traffic only, but if the increase was caused by current events then it is likely that both direct and search traffic will increase.

As the column chart indicates, there is a spike in visitor count for search traffic in the month of July after schema was implemented on 5/29/2020. This means that there was an increase in visitors who visited the website through organic sources, such as search engines. A possible explanation for this spike could be that rich results and better ranking on search engine result pages generated by schema might have established an environment that produced more clicks in the month of July.

Figure 1: For NY Law Help column chart depicting direct and search traffic per month for the year 2020

Metric 3: Session duration

The metric of ‘session duration’ is significant for our analysis, as an indicator of a good match between the site’s content and the user’s intent. The schema implementation aims to increase not just the volume of search traffic to legal help websites, but good matches to the person’s actual needs. If there is an increase in session duration, then this might suggest that the website is attracting more people who find value in the website’s information and are willing to spend more time engaging with it. The visitor session duration metric can also tell us if an increase in visitor count is due to the website being discovered by people who really need legal help from that state, or if it is due to the website being suggested to people who click on it but then realize it is irrelevant to their situation.

For Idaho Legal Aid, we saw a very drastic increase in page views per session for visitors who were on the website for 30+ minutes. This might suggest that the website is attracting more people to whom the website’s content is relevant.

At first glance, it seemed that Legal Aid Hawaii also enjoyed an increase in visitor count, but it turned out that the gain was due to an increase of visitors who stayed for 0–10 seconds. These visitors are more commonly known as bouncers. Law Help NY also saw an increase in bouncers, but there was also a slight increase in the number of visitors who visited for 10 seconds — 10 minutes.

Figure 2: The Idaho Legal Aid website saw an increase in pageview per session after the schema markup implementation.

Metric 4: Click-through rate (CTR) and search ranking

The last metric that can be useful is the click-through rate and search ranking. Indeed, this might be the most appropriate metric to evaluate the markup’s effectiveness, since Google communications claim that schema markup will improve search ranking. Of course, whether that improved search ranking means that more people will click on the search result and interact with the website is a step removed from improved ranking. Florida Law Help had incorrectly implemented their Google Analytics, but they had provided more extensive Search Console data than the other websites. The click-through rate generally increased after schema implementation. Search rank was on the decline until the markup was implemented — then, the website rank steadily started to improve.

Figures 3 and 4: click-through rate and search ranking improvement for Florida Law Help after the schema markup implementation.

Searchers for the organization versus the issue

Two good indicators for whether schema would be effective for a given site are: 1) the proportion of search traffic to direct traffic and 2) the proportion of clicks after the user searches the exact name of the organization. We chose these indicators because markup would only affect a person’s online behavior if they were searching help through a search engine instead of going to the website directly. Also, markup-driven increased search placement or rich snippets are more likely to affect a user’s chances of clicking on a website if they were looking for general legal advice. Users who were already looking for a specific organization’s website will look for and click on the organization’s website regardless of whether there’s a rich snippet or higher placement.

Around 50% of Legal Services of Northern Florida’s traffic was direct, which means that most visitors of their website never got to see the rich snippets at all (since schema implementation adds rich snippets to search results). For both LSNF and Hawaii, 34% and 18% of visitors respectively had typed in the exact name of the organization in the search bar. This suggests that a good portion of visitors to these sites were already looking for the organization’s website, so a rich snippet or other special search engine treatment would not increase their chances of clicking on their results.

Comparatively, only 5% of Idaho’s visitors had typed in the exact name of the organization. Since most of Idaho’s visitors are people searching more generic help terms (like “tenant eviction help idaho,” instead of “idaho legal aid”), the rich snippets can really help draw attention to the site. A legal aid group that wants to increase its search traffic could use schema markup to connect with people searching for their problem issues, rather than for a local legal aid group.

Figure 5: LawHelp Minnesota’s traffic source is mostly search traffic. This signals potential for schema markup implementation.

Confounders of pre- and post-markup search analytics

We recognize that other factors aside from the application of schema markup may have affected the websites’ analytics.

One set of confounding variables are content and technical changes on the websites we are working on. For example, session duration could have also increased due to technical changes on a website. A more complex layout, for example, often means that the website is difficult to navigate. This could be a potential factor that increases the session duration. After a check-in with the public interest groups, we found out that none of the layouts have changed since the implementation of the markup and we could therefore proceed with using the metric in our analysis.

A second set of confounders are around seasonal changes. People’s legal help situations may differ throughout a given year, rising and dipping with different seasons and regular events. Financial legal needs may spike around tax season in the early months of the year, or during the holidays when more money is being spent. Educational problems may rise before and during the first months of a school year. Search traffic may be higher in certain times of the year, and thus there might be changes in search traffic during the analysis period that is based more on changing legal needs than the markup intervention.

A more difficult set of confounding variables emerged with COVID-19. The health and economic emergencies led to both new legal protections, and more people facing legal help situations like needing to file for unemployment benefits, avoid eviction for nonpayment of rent, deal with domestic violence threats, and deal with contracts affected by pandemic restrictions. These new protections and problems likely changed people’s search behavior. We might assume that the pandemic increased the levels of searches for legal help, and also introduced new types of searches (like, ‘is there an eviction moratorium in my state’ or ‘how do i claim my stimulus check’). This changing search behavior presented a major confounding variable for our intended pre-/post-study of the markup’s effects on legal help search.

At first, we thought that we would wait out the emergency period, and run the intervention once it ended. Once we realized that the emergency would be continuing indefinitely, we considered how to incorporate the data fluctuations into our analysis. We adjusted our evaluation, to look not only at pre- and post-markup metrics (both in the year 2020), but also comparing the same time period to the previous year. This method allowed us to spot fluctuations that were likely caused through changed search behavior relating to the pandemic, as opposed to seasonal confounding variables that would occur each year. Moreover, as we noted above, we decided to use multiple website analytics metrics in addition to the visitor count, which would be more susceptible to fluctuations due to current events, such as Covid-19.

We tried to use these techniques to minimize the effects of Covid-19 on our data analysis. But there is no guarantee that we managed to fully eliminate the impact that Covid-19 had on the data analysis, and we recommend that future studies in a more ‘stable’ year will be needed to measure markup’s impact.

Search Engine Results Page Audit, round 2

After implementing the schema markups on our partners’ sites, we conducted a second round of a SERP audit. This second audit was to determine if, after the markup implementation, we saw any differences in how public interest legal aid sites’ placements. We compared the first SERP audit’s results with the second’s, to see if overall three was any change in domain types, and for specific websites’ performance.

One change we observed was a decrease in the amount of .org domains in Northern Florida-based searches.

Figure 6: Search Audit, Round 2: Florida Domain Suffix percent breakdown

We had put markup for two legal help websites with .org domains that try to serve users with legal help queries: https://www.lsnf.org/ ; https://floridalawhelp.org/. Our assumed outcome is that these .org sites would appear more frequently based on the markup, intervention but this did not occur. Interestingly, there was an increase in .gov links that appeared in the results. This was most likely not due to our schema intervention, though. Our markup was only added to organizations at .org domains, and not with .gov domains. The increase in .gov links could be explained by the Covid-19 information and help pages set up by governmental institutions. Their improved website offerings may have increased their search placement.

Figure 7: Search Audit, Round 2: Hawaii Domain Suffix percent breakdown

Hawaii’s second SERP audit illustrates a similar trend. There is a slight increase in .gov domain extensions, but as noted above, the stakeholders that participated in this research project did not have a .gov domain extension. The slight increase in both .gov and .com domains can potentially be explained by the Covid-19 pandemic that created a surge in both searches as well as the creation of help and information pages regarding the pandemic.

Figure 7: Percentage of non-US domain names

There also hasn’t been a significant change in the percentage of non-US domain names. This percentage seems to be nearly the same in November as in January of this year.

The lack of development and changes in the search audits pre- and post-schema markups may be explained by two factors. The first factor is the limited duration of the schema implementation on the websites. As noted earlier in this chapter, most stakeholders implemented the markup in the early summer months. There is no clear explanation from search companies or research data on how long it takes before the effects of the markup can be seen on the result pages of search engines. It might take several more months before we can see the markup being recognized and understood by the search engine crawlers and subsequent ranking algorithms. The markup merely speaks to any crawlers about what is on the page, but there is no documented, predictable process about how the search engine teams and algorithms crawlers take the markup information and use it to affect the search results. There is also the possibility of advocacy to technology companies, to have their search teams pay particular attention to this markup so that they track and evaluate internally how well they are serving legal help searchers.

The second factor is more technical. The schema markups have mainly been implemented on the mainpage of the organizations. However, it might be possible that the benefits of schema are mostly gained when the markup is implemented on the specific pages of the organization’s website. For example, describing that Legal Services of Northern Florida is a legal aid organization that serves the Pensacola and Jacksonville jurisdictions on the main page may not be sufficient. Our first round of markup interventions, focused on organization markup, may not be nearly as effective as issue- and service-oriented markup. Future interventions and analysis should concentrate on adding markup to their specific hotline, guides, and other content on specific issues — like for unemployment, veterans, landlord-tenant, or domestic violence. The main page often embeds general information about the organization, but that’s not necessarily what a user needs. Users are often looking for information on a specific problem and are thus more likely to click on a link on a search engine results page that provides a snippet with specific information. They would be searching for their problem, and not for ‘legal aid group near me’. This means that in future iterations of this project, more attention has to be given to schema markups for specific legal information pages.

Did the Schema.org intervention change the legal help sites’ performance?

Because of the confounders, particularly around COVID-19’s effects on search behavior, we cannot make strong conclusions about schema markup’s effects on traffic to public interest sites. We can use the pre- and post-markup analytics to observe some changes that indicate for some sites experienced higher numbers of visitors and improved click-through rates. But not all sites experienced these increases. This 2020 analysis suggests that there might be value in schema markup to increase search rank, but marking up at the organizational level does not produce a ‘markup bump’.

Speaking with our legal aid practitioners and search engine contacts about these results, some alternate proposals emerged. In future interventions and evaluations, there might be a higher increase in traffic if more of the sites’ individual webpages and services are marked up (not just the general organization). In this way, their specific guides, articles, FAQs, hotlines, and clinics on issue areas like renting, debt, domestic violence, etc. may be communicated to search engine bots. These bots can recognize that these individual pages have content that can help people searching for problems in this area, and then do a better job in matching searchers to this specific page (not the homepage) that can help them with their query.

The results indicate that there might be value for some sites in using markup to increase traffic, it is important to look at the schema markups in more relative terms. Is this intervention, of developing and applying structured data on a website, meaningful enough for wide application in the public interest legal help community? How much does the schema markup improve the search engine results and increase its amount of targeted visitors?

Exact understanding of markup’s impact is difficult to define. Google’s search team and its algorithm do not list conclusively how their algorithm uses schema markup in its search results, and how this differs for particular areas of questions (like for legal help). Even though Google encourages sites to use markup to improve their search placement, there are many other search engine optimization techniques that sites may use, such as mobile-responsiveness of the websites, user-friendly content, search experience of individual users, page-loading times, preventing spam on webpages, etc. Even if a legal aid site has markup, other sites who are using other SEO techniques may still place above them. Implementing schema may increase the likelihood that a site’s content may appear in rich Google search results or better targeting of results to user’s queries, but does not guarantee either.

In the future, Google’s search engine may treat the sites differently if the markup specifies that they are authoritative — with a government designation, public interest credentials, or another standard that is set as authoritative. This would depend on the legal community’s ability to decide what a marker of authority could be, and then discussions with search companies to make them aware of these markers and why they should be considered in their algorithm. For example, a coalition of courts and legal aid groups could identify what makes their sites more authoritative and beneficial to people seeking legal help. They could agree on how to use schema markup terms to communicate their public interest status. Then they could convey this work to the search companies, and attempt to inform them about how they are using markup to designate their authority and why the search algorithm should pay regard to this part of the markup.

Image: An annotated version of our schema markup, that was revised after our initial implementation, to better allow for building onto it with individual pages + articles marked up as well as the main organization and legal aid services.

When we are discussing future work and next steps, it is important to realize that the schema markups are just one part of a larger set of SEO techniques and search engine policy decisions. Markup will not be the sole solution to improve search results, create rich snippets and other special treatment on the results page, and drive more traffic to public interest legal help sites. Equal attention has to be given to creating user-friendly content, easily accessible web layouts, fast loading webpages, compatibility with various devices and browsers, etc. Last but not least, another important factor is setting up partnerships with tech companies. These companies can not only provide public interest groups with the correct information on how to adjust their websites and markups, but there is opportunity for more conversation on why there is a need for legal help snippets and better search engine page results. It is only through a combined effort by both parties that we can increase the accessibility of online legal information.

A big thank you goes out to our research assistants, Yue Li and Julia Park, and all of the wonderful pioneering stakeholders who participated in this research project: thank you!