FEATURES – TECH TALK Winter 2019

TECH TALK
Anatomy of an EBS Upgrade typography
James J. Morrow, BlueStone Solutions Group, Inc.
Even though every upgrade and every platform migration is different, they all distill down to the same basic steps. The differences between them tend to arise from the combinations of platforms, versions and modules, as well as the variances in “what’s current” from Oracle at the time of the upgrade. As always, you should refer to the Oracle-provided documentation when planning your project.
Where to Begin?

As with many things Oracle, you may find yourself piecing together several documents. Start with the E-Business Suite (EBS) upgrade manual (available on http://docs.oracle.com), which will provide a general guide to the upgrade process for EBS. You can also find most of the additional information you need at the EBS R12.2 Information Center on My Oracle Support [MOS 1581299.1]. If you need to upgrade the database and/or do a platform migration, follow these documents. If you’re doing a platform migration, the only things that you’re actually migrating are the database and any customizations. Odds are, any of your customizations are platform independent.

Versions and Support Timelines

When delivering the first iteration of an upgrade, we usually want to go as current as possible. For most of us, that would mean EBS 12.2.9 and RDBMS 19c. Certification of RDBMS 19c was announced in September 2019. Remember, EBS 12.1.3 is under Premier Support until December 2021 and the fee waiver for RDBMS 11gR2 and 12cR1 is in effect until December 2020.

Strategy and Method

Map it out. For each iteration, create a RUNBOOK. During the initial iteration, the RUNBOOK is a scripted list of steps/commands that you intend to use. Once you’ve put together the RUNBOOK, make a copy called the TRANSCRIPT where you will record any errors, failures, fixes or additional steps as you proceed through the iteration. You should also keep track of timing information for any significant steps.

An upgrade is an iterative process. When preparing for your next iteration, take the transcript that you produced during the previous one, remove unnecessary details, and use that as the basis for the runbook for the next iteration. As you’re preparing your next runbook, look for adjustments to your approach so that errors can be corrected in advance.

This approach also allows you to re-sequence your upgrade steps. Consider which steps need to be performed during your “blackout window” and which steps can be performed in advance. These “out of band” steps are generally ones that don’t require access to the database. Things like installing software, merging patches, patching the database $ORACLE_HOME, or any of the tools patches, can all be done in the days or weeks prior to your actual production upgrade, thus dramatically reducing any required downtime.

Up, Up, Upgrade arrow
Decisions to Make
New Hardware/Operating System

An EBS upgrade is the perfect time to consider a hardware refresh and operating system upgrade. One fantastic benefit here is that you have a built-in fall back to your upgrade process because the old system can remain untouched. If you are moving to new hardware, be sure to perform a test upgrade on the new hardware prior to go-live. This will give you reliable timings, but it will also allow you to test any touchpoints with the operating system (directories, permissions, printers, etc.) before the go-live.

Platform Migration

The platform that you’re running on is a combination of the base operating system (Solaris, or Linux, for example) and the processor (SPARC, x86). Simply moving to new hardware or a new version of the operating system isn’t really a platform migration. For the purpose of this document, a “platform migration” is one that would require some sort of transformation. This transformation is usually because they use different libraries/binaries and have incompatible database datafile formats.

DECISIONS TO MAKE

  • New Hardware/Operating System.
  • Platform Migration.
  • Database Upgrade.

If it is available, and you’re doing a platform migration, then what technology do you plan to use for the database platform migration? Depending on platform and version, you may be able to use Transportable Database, Cross-Platform Transportable Tablespaces, DataPump, or, you may be forced to use old-fashioned export/import.

Database Upgrade

You will almost certainly want to upgrade the database to 12cR1 or 19c (current) at some point during this project. The question is, do you want to do the database upgrade “out-of-band” or during your main upgrade window? The answer, as always, is “it depends.” If you’re doing the upgrade “out-of-band,” then that means you will be running your current system on the new database version for some time. As a result, you will want to treat the database upgrade as a separate effort and test accordingly. The degree of end-user testing for a database upgrade can be significant because of the impact that changes to the optimizer might have on the performance of custom code. However, if you’re tight on time, an “out-of-band” upgrade would allow you to reduce your outage for the EBS upgrade by a few hours.

If you’re going to upgrade the database separately, you need to ensure that the database version you’re upgrading to is certified for your current platform, operating system version and EBS release (and that any interoperability patches have been applied. However, if you’re upgrading the database along with EBS, this becomes less important as you won’t be running the old version of EBS against the new database.

While the 19c database has been certified with EBS 12.1.3 and 12.2, many components/features of 19c still have not been certified. Additionally, if you’re planning a platform migration, Transportable Tablespaces and Transportable Database haven’t been certified with 19c.

Prepare Target
Install Software

First and foremost, prepare the operating system. Make sure that any packages and operating system parameters are set. You can find the particulars in the installation guide and the READMEs for your particular operating system.

Once the operating system has been prepared, the next step is to build a staging area. Here, we are extracting the various 12.2 installation CDs and running the buildStage.sh script. You will also want to patch the startCD at this time.

You will then need to run the RapidWiz tool to install the software on the dbTier and the appsTier. Because we’re doing an upgrade, we can choose to create an “upgrade filesystem” that doesn’t lay down and configure a database. One thing to note here is that the current startCD will lay down the Oracle RDBMS 12c software. If you’re planning to use 19c, you will need to install that independently.

Apply Patches

At this time, you should also apply any database-level patches that are on your list. These include any patches recommended for EBS (the interoperability document will contain a list), the latest PSU (or CPUs), as well as any recommended or “high priority” patches you may have already identified. Remember that you don’t need a database yet, and, by applying patches to the ORACLE_HOME before upgrading the database, we can avoid some redundant steps.

Here, it’s important to note that, with EBS 12.2, we now have a tool called the EBS Technology Code-Level Checker (ETCC), which will scan your database ORACLE_HOME (and your appsTier Technology Stack). ETCC is available as patch 17537119. Unlike most patches, the ETCC patch is updated periodically (at least quarterly) without changing the patch number. The ETCC tool will tell you which EBS recommended patches are missing and will even provide you with a patch number that will allow you to download all of them in a single bundle. (You will still need to apply them individually).

You can also patch some things on the appsTier nodes in advance! Apply patches to each of the appsTier $ORACLE_HOMEs as part of your software installation process (and before your blackout window). Again, ETCC can be your guide here. You can apply any patches that don’t require a database.

If you are performing a platform migration as part of your upgrade, you can perform any preparatory steps (like creating an empty database to receive an import) at this time.

Pre-Upgrade Steps (Perform in EBS 12.1)

In our example, we’ve chosen to perform the database upgrade as part of the EBS upgrade outage. We’re also moving to new hardware. Because we haven’t done anything significant (aside from the pre-upgrade steps) to the source environment, we can simply bring the old environment up if we decide to pull the plug.

Before we start the actual upgrade, there are functional pre-upgrade steps that will need to be performed before you do much to the source system. The module-specific pre-upgrade steps for upgrade from any R12 release to R12.2 are straightforward and tend to be things performed as a normal part of your system lifecycle. These steps are well documented in chapter 4 of the upgrade manual. It’s important to note that there is no “TUMS” script for the R12.2 upgrade.

Begin the Upgrade
PHASE ONE: Blackout Window Begins

Here, users are off the system and we shut down the system and begin our blackout window.

In EBS 12.1, disable any database or EBS password complexity rules to avoid problems during the upgrade. We will enable them again on the other side. Make sure that all module passwords are set to known values. This includes APPS, APPLSYS and SYSTEM. Reset APPLSYSPUB and GUEST back to their default values.

There are also a few pre-upgrade steps listed in the upgrade manual that need to be performed by the DBA (disabling triggers, etc.). You will also want to enable maintenance mode at this time.

We also need to copy our datafiles over to our target database and begin the database upgrade. For simplicity, we’re going to assume here that your filesystem layout is unchanged. Copy the datafiles to the new server and upgrade the database according to the MOS note appropriate for your versions and platform. Keep in mind that we’ve already applied any database patches relevant to EBS.

PHASE TWO: Upgrade EBS to R12.2.0

Next, we’re going to apply the latest AD Consolidated Upgrade Patch (CUP) to the 12.2 ${APPL_TOP}. This is done by merging CUP11 with any additional AD patches (Mentioned in MOS 1320300.1) and applying the merge with the same adpatch tool we’re all familiar with.

We’ll also apply the Consolidated Upgrade Patches and some other preinstall patches (MOS 1448102.2) (using adpatch preinstall=y).

Next you will need to merge this driver with the u101024646.drv driver. The command to merge the preinstall driver with the upgrade driver is:

cd ${AU_TOP}/patch/115/driver
admrgpch -d . -preinstall -master u101024646.drv

Apply the merged patch using adpatch and run RapidWiz to configure the instance.

PHASE THREE: Enable Online Patching

Now we come to the interesting part. We need to enable Online Patching. There are a number of tools involved here. Among them, the Online Patching Readiness Reports and the Global Standards Compliance Checker [MOS 1531121.1].

Run the online patching readiness reports. Review the output and correct any problems. Then run the reports again (to ensure that the identified problems are resolved). There will be a significant number of “false positives” coming from EBS components that haven’t yet been upgraded; you can ignore those. You will also notice that many of your custom objects also fail the compliance check. Over the course of your upgrade project, your development team should be correcting any issues. We can ignore these for now because the corrected code will be migrated into production during the go-live.

Once you’ve remediated the online patching items, apply the online patching enablement patch and, if successful, re-run the reports. These can help to guide you through your code remediation later in the project.

From this point forward, patches will be applied using the adop tool (rather than our old, familiar, adpatch).

PHASE FOUR: Upgrade to R12.2.9

Here, we branch off to the R12.2.9 README [MOS 2495027.1]. We’re going to apply the latest AD Patch (AD.C.Delta.11), latest TXK patch (TXK.C.Delta.11, and the critical AD & TXK patches [MOS 1617461.1]. These patches are applied using adop hotpatch=yes. Once these have been applied, you will also need to run autoconfig, re-source your environment and re-run the ETCC tool, which will record version information regarding your Database ORACLE_HOME and the appsTier Technology Stack. Future runs of the adop tool will check the ETCC information and let you know if you’re out of date.

Pre-Upgrade Steps

  1. Start the Blackout Window.
  2. Upgrade EBS to R12.2.0.
  3. Enable Online Patching.
  4. Upgrade to R12.2.9.
  5. Patch Current.
  6. Complete Post-Upgrade Tasks.

After applying AD.C.Delta.11, the ${PATCH_TOP} environment variable is added pointing to a directory under ${APPL_TOP_NE} where patches are staged before they are applied. Do NOT clean up this PATCH_TOP directory until after running FS_CLONE or PREPARE [MOS 2348982.1].

Our next step is to apply the Consolidated Seed Table Upgrade Patch and R12.2.9 patch and complete the patching cycle.

PHASE FIVE: Patch Current

So, the next question that usually comes up is which patches should be applied after the upgrade. My practice, at least early in the project, is to apply any and all family packs, as well as the most recent security patches. Through the duration of the project, since the system will be thoroughly tested, this should relieve any fears. As you get closer to your go-live date, the only patches you should add are security patches and bugfixes that arise during your testing.

PHASE SIX: Post Upgrade Tasks

As part of your post-upgrade, the system will be locked down and you will need to respond to the Secure Configuration Console [MOS 2393248.1]. You aren’t required to correct the security items that are identified, but you will need to acknowledge them to unlock the system.

At the start of this process, we changed the key passwords (APPS, APPLSYS, etc.) to default values. Because of this, you will want to change them to production-quality values. You’ll also need to perform some more familiar maintenance tasks like updating the Java Plugin and signing your JAR files.

Phase 6: Post Upgrade Tasks
Finishing up the dbTier

Once you’re done with upgrading and patching the appsTier, you’ll need to fully AutoConfig-enable the dbTier $ORACLE_HOME. The first step will be to run admkappsutil.pl and deploy a new $ORACLE_HOME/appsutil directory onto your dbTier. Once you’ve done that, you can generate your dbTier context file using adbldxml.pl and then run AutoConfig on the dbTier.

Summary

As you run through your upgrade project, having a well-documented and repeatable process is critical. Use the RUNBOOK/TRANSCRIPT approach to record each iteration and plan out the next until the iterations run smoothly and cleanly. Planning, practice and thorough documentation are the keys to your success.

Author James Morrow

James Morrow has worked in information technology for over 30 years, including 25+ years as an Oracle Applications DBA and Unix Systems Administrator. He has extensive experience in systems architecture, including installations, upgrades and advanced configurations of Oracle’s EBS applications. He has authored or co-authored several papers, including Installing, Upgrading and Maintaining Oracle EBS Applications Release 11.5.10+ and is a frequent presenter at COLLABORATE conferences.

morrow.james@bluestonesolutionsgroup.com
www.bluestonesolutionsgroup.com